Proliferation of unverified AI-generated data is increasingly becoming a big challenge for modern enterprises.

Even as AI is becoming deeply integrated into the workflows and systems, there’s a risk of something called “model collapse”. This essentially occurs when large language models (LLMs) at some point start training on synthetically generated content, creating a feedback loop that could very well jeopardize the very intelligence it’s built on.

And this is why there’s a lot of conversations around building zero-trust data governance. It’s now widely being seen as becoming a strategic framework rather than just keeping it a niche security protocol. Moreover, it provides key checks and filters in the era of AI.

Model Collapse Risks

According to the latest Gartner report, and also as mentioned above, LLMs are typically trained on “web-scraped” data and a variety of sources, including books, code repositories and research papers. Some of these sources already contain AI-generated content, and if current trends continue, nearly all will eventually be populated with AI-generated data.

The report also mentions a 2026 Gartner CIO and Technology Executive Survey wherein 84% of respondents said that they expect their enterprise to increase funding for GenAI in 2026.

Evidently, with enterprises fast forwarding with adoption and investment in AI initiatives, the volume of AI-generated data will significantly grow. This could create a scenario where future generations of LLMs will increasingly be trained on outputs from previous models, heightening the risk of “model collapse,” where AI tools’ responses may no longer accurately reflect reality.

“As AI-generated content becomes more prevalent, regulatory requirements for verifying ‘AI-free’ data are expected to intensify in certain regions,” said Wan Fui Chan, Managing VP at Gartner, in a statement. “However, these requirements may differ significantly across geographies, with some jurisdictions seeking to enforce stricter controls on AI-generated content, while others may adopt a more flexible approach.

“In this evolving regulatory environment, all organizations will need the ability to identify and tag AI-generated data. Success will depend on having the right tools and a workforce skilled in information and knowledge management, as well as metadata management solutions that are essential for data cataloging.”

“Augmentation have been standard practices in machine learning for years, helping expand coverage, balance datasets, and reduce bias. The real risk emerges when data volume begins to replace data quality. As low-quality AI generated content increasingly saturates the web, models are exposed to large amounts of unverified or weakly grounded data. Over time, this leads to the amplification of errors, biases, and oversimplifications. Models may sound more fluent, but they become less reliable and less connected to reality. Rare viewpoints and edge cases gradually disappear, reducing cognitive diversity and increasing the risk of a collapse,” Paul Jebasingh, founder of Two Minute Reports, told Entrepreneur India.

Zero-Trust Data Governance

The Gartner report predicts that the zero-trust data governance model will find more takers in the coming years – “by 2028, 50% of organizations will implement a zero-trust posture for data governance due to the proliferation of unverified AI-generated data.”

“Organizations can no longer implicitly trust data or assume it was human generated,” said Chan. “As AI-generated data becomes pervasive and indistinguishable from human-created data, a zero-trust posture establishing authentication and verification measures is essential to safeguard business and financial outcomes.”

Apurv Agrawal, CEO & Co-Founder, SquadStack.ai explains to Entrepreneur India that the shift toward Zero-Trust Data Governance is both inevitable and overdue. The traditional idea of a trusted perimeter no longer holds when data flows continuously across cloud platforms, APIs, vendors, and AI systems. In an AI-native world, data itself becomes the attack surface.

Zero-trust flips the model from “trust by location” to “trust by verification.”

Every dataset, signal, and source is continuously authenticated, versioned, and governed, regardless of where it originates. This architecture makes sense because modern organizations are no longer defending networks; they are defending decision systems. When AI models are trained, fine-tuned, and updated in near-real time, governance must move upstream into the data layer itself.

“The prediction by Gartner reflects a broader realization: as AI systems increasingly influence revenue, risk, and customer outcomes, organizations cannot afford implicit trust in their data pipelines,” Agrawal added.

According to 3Cubed founder Shammik Gupta, zero-trust makes sense because the boundary has moved. Data now flows through many tools, vendors, and AI systems before it reaches a decision-maker. You can’t assume inputs are reliable just because they are inside the organisation. Zero-trust simply means checking where information comes from and being clear about who is responsible before acting on it.

“The real risk is not technical failure. It’s bad judgment. When AI-generated content feeds back into business systems, decisions can start reinforcing assumptions instead of reflecting reality. Edge cases get missed, risks are softened, and leaders gain false confidence,” Gupta added.

The Failsafe

Gartner suggests that organizations should consider several strategic actions to manage the risks of unverified data that includes appointing an AI Governance Leader.

“Establish a dedicated role responsible for AI governance, including zero-trust policies, AI risk management and compliance operations. This leader should work closely with data and analytics (D&A) teams to ensure both AI-ready data and systems capable of handling AI-generated content,” it said.

Similarly, it stressed the need to foster cross-functional collaboration, leverage existing governance policies, and adopt active metadata practices.

Agrawal adds that there are architectural safeguards that help preserve model integrity. One is strict data provenance tracking, where every training signal is tagged by origin, confidence level, and human verification status. Another is temporal isolation: separating fresh, human-generated data from legacy or synthetic-heavy corpora during retraining cycles.

“A third, often overlooked failsafe is reinforcement through real-world interaction. Models grounded in live, outcome-based feedback, such as customer conversations, transactional results, or human corrections, are far more resilient to synthetic drift than models trained primarily on static internet data,” he said.