RAG Poisoning & Knowledge Corruption

Concept. RAG was supposed to make LLMs safer — ground answers in a trusted corpus instead of the model's fallible memory. Week 12 is where that promise inverts: the retrieval corpus becomes the highest-leverage attack surface in the whole stack. This is OWASP LLM08:2025 Vector and Embedding Weaknesses and the training-time data-poisoning family, studied offensively. The through-line is brutal and repeatable across every paper below: a handful of documents, sometimes one, flips the model's answer — and the classic defenses (perplexity, paraphrasing, dedup) don't catch it.

🎯 Objectives

By the end of this week you can:

• Execute a PoisonedRAG-style corpus injection on a local vector DB and explain the two independent conditions a poison must satisfy to work (PoisonedRAG, USENIX '25).
• Distinguish retrieval-time poisoning (corpus, KG, PhantomText) from training-time poisoning (label-flip, clean-label, backdoors, quantization-conditioned) and pick the right defense for each.
• Craft a poisoned document with vocabulary engineering + authority framing and measure attack-success rate against a real embedding model (Amine Raji walkthrough).
• Explain why graph-structured corpora move the attack rather than stop it — GraphRAG filters naive poisons at indexing, but shared-relation and coreference attacks (GragPoison, UKPA) exploit the graph structure itself (GraphRAG under Fire).
• Reason about why more precision ≠ more safety — a clean model can hide a backdoor that only fires after int8/int4 quantization (QVec).
• Instrument a corpus with canary documents so unauthorized retrieval and exfiltration trip a tripwire (Canarytokens).

The big picture

Poisoning splits along one axis — when the payload enters the model's knowledge — and that axis decides everything about detection.

Retrieval-time attacks never touch the weights; they inject text the retriever will later fetch. Training-time attacks bake the payload into the parameters themselves. RAG is the softer target because there is no retraining, no gradient access, no privileged position required — just write access to (or influence over) a corpus. As soon as an LLM reads any document it did not author, that document is untrusted input, and the same prompt-injection rules from Week 9 apply — except now the injection is persistent and retrieved on demand.

🔑 Frame for the week: a retrieved document is not a fact — it is attacker-reachable input that the model has been told to trust. Treat the vector DB the way you treat a request body: hostile until validated.

The attack surface

1 · Corpus poisoning — the canonical RAG attack

PoisonedRAG is the reference result. Injecting 5 malicious texts per target question into a knowledge base of millions of documents yields a 90% attack-success rate — the LLM confidently emits the attacker's chosen answer (Zou et al., USENIX Security '25). The paper's key contribution is decomposing a successful poison into two independent conditions, each of which you must satisfy:

• Retrieval condition — the poisoned text must rank high enough to be fetched for the target query (win the similarity search).
• Generation condition — once in context, the text must be persuasive enough that the LLM prefers it over any legitimate passages retrieved alongside it.

Black-box variants (no knowledge of the retriever) still work by making the poison lexically resemble the target question; white-box variants optimize the embedding directly. Crucially, the authors tested paraphrasing, perplexity filtering, and duplicate filtering and found all three insufficient — a result every defense in Week 17-18 has to beat.

The trend since is toward fewer documents and harder targets. CorruptRAG drops the count to one. A single poisoned text out-performs multi-doc baselines like PoisonedRAG because it never has to outnumber the legitimate passages — it only has to discredit them, so it also sidesteps the "can the attacker really insert five docs per query?" objection. Its two variants both weaponize recency bias: CorruptRAG-AS wraps the payload in a structured template that labels the real answer "outdated" and the attacker's answer "current"; CorruptRAG-AK frames the wrong answer as a newly-verified freshness update (Zhang et al., SACMAT '26, arXiv 2504.03957). How well that lone document lands is dominated by the victim's architecture, not the attacker's effort — under CorruptRAG-AK, attack success spans 81.9% (vanilla RAG) → 24.4% (a reasoning-augmented pipeline) at near-identical clean accuracy, making the retrieval architecture the single biggest robustness lever a defender holds (Architecture Matters, arXiv 2605.05632).

Graph-structured corpora don't help — they move the attack, they don't stop it. In GraphRAG the corpus is a knowledge graph, and naive document poisons are often filtered out at indexing because the LLM extractor discards incoherent statements — but that same structure is the new attack surface. GragPoison exploits relations shared across queries: because "How do I mitigate Stuxnet?" and "How do I detect it?" both ride the triple Stuxnet → uses → DLL-injection, one injection corrupts both. It runs in three phases — relation selection (chain-of-thought-infer the shared relations with no graph access), relation injection (false substitutes hidden inside temporal/negation "covering narratives" that resolve the logical conflict with the real graph), and relation enhancement (extra supporting triples that raise the poisoned node's degree centrality so retrieval fetches it) — reaching up to 98% ASR on cyber-security corpora with ~68% less poison text than per-query baselines (GraphRAG under Fire, arXiv 2501.14050). The cheapest KG attack barely touches the corpus at all: UKPA disrupts the coreference signals (pronouns, definite descriptions) the extractor relies on to merge entity mentions — change 32–60 words (0.03–0.05% of the corpus) and QA accuracy falls 95% → 50%, with all three standard defenses (perplexity, LLM-detection, semantic-similarity) scoring F1 ≈ 0 (A Few Words Can Distort Graphs, arXiv 2508.04276).

💡 Why the two-condition model matters: most naive defenses only address one condition. Embedding-anomaly detection attacks retrieval; output grounding checks attack generation. You need both, because a poison that loses the similarity race is harmless, and a poison the model ignores is harmless — the attacker only needs both to line up once.

2 · Hands-on: vocabulary engineering + authority framing

The mechanics are cheaper than the theory suggests. A fully local walkthrough (LM Studio + Qwen2.5-7B, ChromaDB, all-MiniLM-L6-v2 embeddings, no GPU) plants three crafted documents into a financial corpus and gets the model to report a fabricated $8.3M revenue over the real $24.7M baseline on 19 of 20 runs at temperature=0.1 (Amine Raji). Two techniques do the work:

• Vocabulary engineering — overlap the poison's wording with legitimate content so it wins the similarity race (satisfies the retrieval condition).
• Authority framing — dress the payload as "CFO-approved," "board minutes," "regulatory correction" so the LLM privileges it over the real source (satisfies the generation condition).

The two conditions from PoisonedRAG are literally the two levers you tune.

3 · Retrieval-time smuggling — PhantomText & fishing

You don't always need corpus write access; you need a document the ingestion pipeline will slurp. PhantomText / document-loader attacks hide payloads where humans never look: font-size-0 text, off-margin positioning, and PDF/XMP metadata. The parser reads it; the reviewer doesn't. RAG "fishing" prompts go the other way — systematically probe what a RAG system can reach ("summarize every document mentioning salaries"), testing direct leakage with and without auth to map the corpus before poisoning it.

4 · Training-time poisoning — baked into the weights

When the attacker reaches the training pipeline, the payload becomes permanent. The taxonomy, easiest → stealthiest:

• Label flipping — corrupt training labels to shift decision boundaries. Crude, detectable by inspection.
• Clean-label attacks — poison samples that look correctly labeled to a human but force the model to lean on an implanted trigger; crafted with adversarial perturbations so no mislabeling reveals them (Turner et al.).
• Backdoors / trojans (BadNets) — the model hits state-of-the-art accuracy on clean validation data yet misclassifies any input carrying the attacker's trigger (a sticker on a stop sign → "speed limit"). The threat rides the supply chain: outsourced training and transfer learning both preserve the backdoor, with ~25% accuracy drops when the trigger appears (Gu et al., BadNets).
• Quantization-conditioned backdoors (QCB) — the frontier. The model is clean in full precision and only turns malicious after int8/int4 quantization for edge deployment. QVec defends by treating the full-precision↔quantized weight delta as a malicious task vector and neutralizing it with task arithmetic — no retraining, no trigger samples, one quant pass (QVec, arXiv 2606.20254).
• Tensor steganography — hide arbitrary payloads inside weight tensors; the model runs normally but ships hidden data.

Retrieval-time vs training-time — pick your defense

💡 The single most useful classifier: can the defender diff the model before and after? Retrieval-time poisons live in data you can inspect and version; training-time poisons live in weights you often can't. That's why RAG poisoning is the cheap attack and QCB is the scary one.

Defenses you can build this week

• Canary documents — plant uniquely-identifiable decoy docs in the vector DB. If a canary ever surfaces in an output, retrieval or exfiltration happened; free generators at Canarytokens. This is your detection layer, wired up properly in Week 17-18.
• Ingestion-time anomaly detection beats generation-time monitoring — in the local test, embedding-anomaly detection alone cut success 95% → 20%, and layered defenses reached ~10% residual (Amine Raji). Catch the poison before it's a vector, not after it's an answer.
• Automate the red-team loop — the promptfoo RAG-poisoning plugin runs redteam poison to generate malicious docs, you inject them, then redteam run scores whether the LLM produced the attacker's intended result. A passing poison test means you're vulnerable. Chain this into the Week-14 harness alongside toolkits like HackAgent.

🔑 The one rule to carry out of this week: every document your RAG retrieves is untrusted input. Sign and provenance-check the corpus, detect anomalies at ingestion (not generation), plant canaries, and never let a single high-similarity document override the rest of the context.

🎯 OSAI exam depth — Exploiting RAG Pipelines (m5)

The sections above nail knowledge-base poisoning (PoisonedRAG, CorruptRAG, vocabulary + authority framing). For a 24h hands-on exam you also have to beat the parts of a real pipeline that quietly kill naive poisons: the retriever, the re-ranker, and the chunker — and then actually do something with the context you control. These three drills close that gap.

5 · Retrieval manipulation & re-ranker abuse

A production RAG stack is not "embed → top-k → generate." It is sparse + dense retrieval → cross-encoder / LLM re-ranker → top-5 truncation → generate, and each stage is a filter you must survive. Empirically, the retriever alone silently drops ~20% of injection attacks before they reach the model, and gradient-token attacks that looked devastating on a "frozen context" collapse to <2% end-to-end success once retrieval and reranking are in the loop (Can It Reach the Generator?, arXiv 2605.28017). Know how each stage is attacked:

• Dense-retriever hijack (white/grey-box). Optimize the poison's embedding, not just its wording. HotFlip greedily flips one token at a time toward the query's gradient; AGGD (Approximate Greedy Gradient Descent) does it far more efficiently and gets 80.9% ASR from a single injected passage on ANCE, and the passage transfers to unseen queries in other domains (AGGD, arXiv 2406.05087). GASLITE produces fluent adversarial passages that were retrieved for 65% of queries vs 5% for the raw malicious text — a >10× lift from embedding optimization alone (GASLITE, arXiv 2412.20953).
• Sparse / hybrid (BM25) abuse. Dense tricks don't help against the BM25 leg of a hybrid index — there you keyword-stuff the exact query terms (classic black-hat SEO for retrievers). Win both legs or the score fusion demotes you.
• Re-ranker abuse (the stage most attacks forget). The cross-encoder / LLM re-ranker rewards locally coherent, answer-bearing passages and punishes gibberish suffixes. It gives surviving attacks a +16.5pp boost but its top-5 truncation costs ~4.6pp — so you must rank in the top few, not just get retrieved (survival study). CRAFT-style attacks fine-tune a generator to emit passages that promote themselves across ranker types (embedding, cross-encoder, and LLM rerankers all transfer) (Led to Mislead, arXiv 2605.01591). Black-box DeRAG finds a gradient-free adversarial suffix that hijacks the evidence set through the ranking stage with query-only access (DeRAG, arXiv 2507.15042), and Topic-FlipRAG nudges opinion/stance rather than facts — useful when the exam target is "make the assistant argue X" (Topic-FlipRAG, arXiv 2502.01386).
• Exam heuristic. LLM-driven prompt-optimization attacks (CORE-reason 53.5%, TAP 40.2%) survive the full pipeline; pure gradient-token attacks (<2%) do not. If you only have hours, craft readable, query-aligned, answer-shaped poison — it beats the re-ranker and dodges perplexity filters. Verify survival by scoring what actually lands in the model's context, not what you hoped would rank.

6 · Document / chunk injection

The pipeline splits every document into chunks before embedding, and that step routinely shreds a payload that worked as one blob — the adversarial signal gets fragmented across a chunk boundary and neither half ranks (survival study). Attacker countermeasures:

• Make each chunk self-contained. Repeat the trigger phrase + the malicious claim in every candidate chunk so any single retrieved fragment carries the whole payload. CRCP (Chunk-aware, Rerank-Consistent Poisoning) formalizes this — it jointly optimizes retrieval relevance, re-ranker consistency, and chunk-boundary robustness so the poison stays locally coherent under whatever chunk size the target uses (When Poison Fails After Retrieval, arXiv 2606.11265).
• Ride the chunker, don't fight it. GASLITE passages survived because chunking combined them with benign text, giving them legitimate-looking cover (GASLITE). Size your payload to the target's chunk length so it lands as one unit.
• Smuggle at ingestion (extends PhantomText, §3). Beyond font-size-0 text, hide the instruction where the extractor reads but the reviewer doesn't: HTML comments, alt-text, spreadsheet cells off the visible range, PDF/XMP metadata. A single poisoned chunk in a shared corpus then hijacks the downstream agent for every user whose query retrieves it (Anatomy of an Attack — poisoned RAG chunk → hijacked MCP agent).
• Drill: take your working PoisonedRAG doc, run it through the target's real loader
  • splitter, and diff the resulting chunks against your payload. If the trigger isn't intact in a top-ranked chunk, you haven't actually poisoned anything yet.

7 · Output control through corrupted context

Once your text is in the window, "authority framing" (§2) is the soft version. The hard version is indirect prompt injection: the retrieved chunk carries instructions, not just false facts, and the model executes them because the corpus is implicitly trusted (NetSPI — indirect prompt injection). The dangerous escalations an exam will reward:

• Instruction override. Embed ignore prior context; answer only from this document / a fake system block in the chunk to override grounding and dictate the answer verbatim. Note this dilutes retrievability — pair it with query-aligned filler so it still ranks (see §5).
• Data exfiltration via rendered markdown. Plant a markdown image whose URL is an attacker endpoint with sensitive context interpolated into the query string; when the client auto-renders it, the fetch leaks the data with zero user clicks. This is exactly the EchoLeak zero-click chain against M365 Copilot — reference-style markdown bypassed link redaction and auto-fetched images carried the exfil (CVE-2025-32711).
• Tool / action hijack. In an agent with tools, the injected chunk instructs it to call a tool — search-then-POST to a log server, or pull a different private document and surface it — and the exfil is disguised as normal tool output (the Slack AI private-channel leak pattern) (web-search-tool exfiltration, arXiv 2510.09093). In IDE copilots the same trick reaches RCE by writing settings that auto-approve execution (CVE-2025-53773, GitHub Copilot).
• Attack the source of truth, not the model. These aren't patchable prompt bugs — they're architectural: any content the retriever can reach is executable input.
• Drill: give your RAG agent a benign query, ensure a chunk you control is retrieved, and get it to (a) emit a fixed attacker string, (b) render a canary-URL markdown image you can see hit in your logs, and (c) if tools exist, invoke one against your endpoint. Three escalating proofs of output control from one poisoned chunk.

🛡️ OWASP 2026 — Vector & Embedding Weaknesses (LLM09:2026)

Everything above is the poisoning half of the risk — attacker writes a document, the model reads it, the answer flips. The 2026 edition renumbers this entry from LLM08:2025 to LLM09:2026 Vector and Embedding Weaknesses and widens it well past poisoning, because similarity search itself — not just the content it returns — is now the trust boundary. The edition's own one-line frame is worth memorizing for the exam: "poisoning makes the system wrong, inversion makes it leak, jamming makes it silent, and access-control failure makes it indiscriminate" (OWASP GenAI — LLM Top 10 for 2026). The unifying property of the three attacks below is that they exploit the geometry of the embedding space, not the model's instruction-following — so none of them require a single malicious instruction in the retrieved text, and every input-sanitizing / prompt-injection defense from Week 9 sails right past them.

8 · Retrieval jamming — the availability attack (no payload)

Corpus poisoning changes the answer; jamming removes it. The attacker inserts one "blocker" document engineered to be retrieved for a target query and to make the LLM refuse or claim it has no information — a denial-of-service on the retrieval layer. The blocker carries no instructions, no false facts; it purely exploits retrieval mechanics plus the model's own safety/refusal behavior, so instruction-scanning and injection filters never fire (Shafran et al., "Machine Against the RAG," USENIX Security '25). The threat model is minimal: query-only, black-box access (attacker doesn't know the embedding model or the LLM) plus insert-only write access to the corpus. Their black-box method generates the blocker by maximizing the semantic similarity between the RAG system's response and a canned refusal string via a surrogate embedding model — a single blocker is enough, and standard LLM safety metrics don't detect it. Defense is the same provenance + ingest-anomaly stack as poisoning (flag a new vector that ranks for a suspiciously wide spread of queries), plus a grounding/answerability check that distinguishes "the corpus genuinely lacks this" from "one retrieved chunk induced a refusal."

9 · Cross-tenant leakage via shared similarity search

In multi-tenant RAG, the fatal ordering bug is that similarity search runs over the whole index first, and tenant ACLs are applied afterward at the app layer. Even when every document is correctly tagged and every call authenticated, a tenant can probe the shared index and read another tenant's data indirectly — result counts, score distributions, and query timing reveal the existence, topic, and approximate volume of the other tenant's documents without ever returning a single one of their chunks. This compounds badly with embedding inversion: because leaked vectors are recoverable to source text, a vector-store auth bug is strictly worse than the same bug in a normal database. Real vector-DB CVEs sit adjacent to (but out of scope of) the geometric risk — CVE-2025-64513 (Milvus, forged sourceID header bypassing auth, CVSS 9.3) and CVE-2025-69286 (RAGFlow, predictable token derivation → account takeover, CVSS 9.3) (OWASP GenAI — LLM Top 10 for 2026). Fix: enforce tenant scope inside the index query, validated server-side — never as a post-retrieval filter. For high-sensitivity data use physically separated per-tenant indexes, apply ACLs at the chunk level (a public doc can hold one confidential paragraph), and treat embedding/similarity endpoints as first-class authenticated APIs with per-tenant rate limits.

10 · Membership inference — the index as an oracle

Sometimes the attacker doesn't want the content, only to know whether a specific document exists in the index — a medical record, an HR complaint, a legal filing. That fact alone can be sensitive. If the app returns raw similarity scores or distances to the client, the index is a direct membership oracle with no LLM in the loop — a low score for a document you hold means "not indexed," a tight match means "present." Even when only generated answers are returned, membership still leaks: submit partial or masked versions of the target document and measure how accurately the system reconstructs the missing spans — if the full document is retrievable, the masked-token prediction is markedly better (Anderson et al., "Is My Data in Your Retrieval Database?", arXiv 2405.20446). The underlying reason a leaked vector is a real breach is embedding inversion: Vec2Text reconstructs up to ~92% of a short 32-token input from its embedding alone via iterative correction, so "only the embeddings leaked" is not a safe-harbor classification (Morris et al., "Text Embeddings Reveal (Almost) As Much As Text," EMNLP 2023). Defense: never return raw scores or distances to clients; add noise / diversification at the ranking layer; rate-limit endpoints that could be queried as oracles; and treat vector-store backups at the same sensitivity tier as the source documents.

🎯 Exam frame — the two conditions, mapped to ATLAS. Retrieval-time poisoning succeeds only when both hold at once: the poison is retrieved (the geometric condition — it lands near the query in embedding space) and it steers the response (the generation condition — the model prefers it). Defenders can break either leg, which is why ingest-anomaly detection (attacks retrieval) and output grounding (attacks generation) are complementary, not redundant. MITRE ATLAS catalogs this class as AML.T0070 (RAG Poisoning) under the Persistence tactic (MITRE ATLAS AML.T0070). Jamming and membership inference are the same embedding-geometry surface turned toward availability and confidentiality instead of integrity.

📇 Poisoning technique reference

The lesson above is what to learn. This is the catalog behind it. Folded by default; expand for the technique-by-technique detail.