Generative AI has fundamentally changed how software is built and how information is consumed. As organizations rely more on Large Language Models (LLMs), a new risk has become increasingly clear: AI systems can generate confident but incorrect outputs that users trust and act on.
In the OWASP Top 10 2025 for LLM Applications, this risk is formally identified as LLM09: Misinformation.
LLM-generated misinformation can lead to real security and business impact, including compliance violations, faulty decisions, and downstream system failures. This blog explains what LLM09 means in practice, why it occurs, and how security teams can reduce this risk effectively.
What OWASP Means by LLM09 – Misinformation
One common misunderstanding is to equate LLM misinformation with disinformation. The two are very different.
Disinformation involves intent, deliberately spreading false information to mislead. LLM09, as defined by OWASP, focuses on something far more subtle and dangerous: models generating false, misleading, or nonsensical content that appears authoritative and credible.
In most cases, the model is not behaving maliciously. It is behaving exactly as designed.
LLMs are probabilistic systems. They predict the next most likely word based on patterns learned during training. They do not verify facts, reason about truth, or understand consequences. When an LLM responds with absolute confidence but incorrect information, it creates what many teams discover too late, a trust trap. Users assume accuracy because the answer sounds correct, and systems act on outputs that were never validated.
That gap between confidence and correctness is where misinformation becomes operationally dangerous.
Why LLMs Produce Convincing Falsehoods
To control misinformation, teams need to understand where it comes from. These failures are not random; they follow predictable patterns.
1. Hallucination: The Primary Engine
Hallucination is the most common driver of misinformation. When an LLM lacks sufficient information, encounters ambiguity, or is pushed outside its training scope, it rarely admits uncertainty. Thus, filling the gap with a statistically plausible answer.
The real risk lies in “how” these answers are delivered. Hallucinated outputs often use the correct tone, structure, and terminology like legal language that sounds precise, medical explanations that feel clinical, or code that looks syntactically valid. Briefly, there is nothing that signals the information is fabricated.
2. Training Data Limitations
An LLM can only reflect what it has seen. If its training data contains outdated material, biased viewpoints, or factual gaps, those weaknesses are reproduced with confidence. The model does not know that a regulation has changed; a vulnerability has been patched, or a library has been deprecated, unless it is explicitly grounded in up-to-date sources.
3. Lack of Grounding to a Source of Truth
When an LLM operates without access to verified documents or live data, it relies entirely on internal model weights. In domains like cybersecurity, finance, or law, where accuracy depends on current information, this becomes a serious liability. The moment training ends; the knowledge begins to age.
The Human Factor: Over-Reliance as a Risk Multiplier
Over-reliance used to be listed as its own OWASP category. In 2025, it was folded into LLM09 for a reason: misinformation rarely causes damage on its own. It becomes a security incident when people trust it without question.
This shows up in familiar ways:
- A developer copies AI-generated code straight into production.
- A lawyer cites a case that never existed.
- A customer support chatbot commits the company to a policy it does not offer.
Because LLMs respond fluently and confidently, users often skip verification. What starts as a probabilistic suggestion quickly becomes a perceived source of truth. At that point, failure is no longer theoretical; it is operational.
How Misinformation Appears in Real Systems
Misinformation does not surface randomly. It tends to cluster around specific use cases.
1. Package Hallucination and Supply-Chain Risk
One of the most serious risks for application security teams comes from AI-assisted development. LLMs regularly suggest software packages or libraries that sound legitimate but do not exist.
Attackers have learned to exploit this behavior. They monitor common hallucinated package names, register them on public repositories, and seed them with malicious code. When a developer installs the package suggested by the model, the compromise happens silently, inside the build pipeline.
Explore the key steps to prevent LLM supply chain risks.
2. Customer-Facing Accuracy Failures
When LLMs are embedded in chatbots or virtual agents, they effectively speak on behalf of the organization. A hallucinated refund policy, incorrect safety guidance, or fabricated entitlement can quickly escalate into legal disputes and regulatory scrutiny.
3. Fabricated Expertise in High-Stakes Domains
In healthcare and legal environments, being “almost right” is not acceptable. LLMs have been shown to invent legal precedents, misinterpret clinical guidance, or suggest treatments based on linguistic patterns rather than evidence. These failures carry real risk, even when no attacker is involved.
Mitigating LLM09: Going Beyond Better Prompts
Reducing misinformation in LLM-based systems requires layered controls rather than a single fix. The goal is to limit hallucinations, detect errors early, and prevent unchecked outputs from being treated as authoritative.
1. Ground Responses Using Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation helps control misinformation by anchoring responses to trusted data sources. Using curated documents or internal knowledge bases ensures outputs remain aligned with verified information, reducing the likelihood of unsupported or outdated claims.
2. Improve Output Quality Through Fine-Tuning
Fine-tuning and embedding-based approaches help align models with specific domains where accuracy matters. Techniques such as parameter-efficient tuning and structured reasoning prompts improve relevance and consistency, lowering the likelihood of misleading outputs in specialized use cases.
3. Enforce Cross-Verification and Human Review
LLM outputs should not be accepted at face value in high-impact scenarios. Cross-checking responses against reliable sources and involving trained human reviewers helps catch inaccuracies before they propagate. Human oversight is especially important where legal, medical, financial, or security decisions are involved.
4. Apply Automated Validation Controls
Automated validation mechanisms provide scalable protection by checking outputs against predefined rules, constraints, or reference data. In sensitive environments, these controls should flag or block responses that fail accuracy or safety checks.
5. Communicate Risks and Limitations Clearly
Users must understand that LLMs can produce incorrect information. Clearly communicating model limitations and the potential for misinformation reduces blind trust and encourages responsible use of AI-generated content.
6. Maintain Secure Coding Practices
When LLMs assist with software development, their output should be treated as untrusted input. Standard code reviews, dependency checks, and security testing help prevent vulnerabilities introduced by incorrect or hallucinated code suggestions.
7. Design Interfaces That Discourage Over-Reliance
User interfaces and APIs should make it clear when content is AI-generated, highlight accuracy limitations, and define intended use cases. Thoughtful design reduces the risk of users mistaking probabilistic outputs for guaranteed truth.
8. Train Users to Evaluate AI Output Critically
Training users on LLM limitations and verification practices is essential. In domain-specific contexts, targeted education helps users assess whether an output is reasonable, complete, and suitable for the task at hand.
