RAG Explained: Why Your Chatbot Still Makes Things Up

Two of the largest legal research vendors on earth told their customers that retrieval-augmented generation had solved hallucination. Thomson Reuters said its tool avoids them. Casetext said RAG eliminates them. In 2024 a team at Stanford ran the first preregistered empirical test of those claims and found the tools invented things between 17% and 33% of the time.^[2]

I keep coming back to that study, because the gap it measures is not a gap between a good product and a bad one. It is a gap between what people think RAG does and what RAG actually does. Almost every team I talk to has the same mental model: bolt a search index onto a language model and the model stops lying. That is not the deal. The deal is much narrower, and understanding exactly how narrow it is turns out to be the difference between a retrieval system you can trust and a demo that embarrasses you in front of a customer.

What the original RAG paper actually promised

Retrieval-augmented generation is not a product category. It is a technique from a paper published on May 22, 2020 by Patrick Lewis and eleven co-authors at Facebook AI Research, University College London and NYU.^[1] The idea: give a language model two kinds of memory instead of one. Parametric memory is what the model learned during training and carries in its weights. Non-parametric memory is an external index it can look things up in, in their case a dense vector index of Wikipedia queried by a neural retriever.

Read what they claimed. RAG models “generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.”^[1] More factual. Not factual. That is a comparative, and the comparison is against a 2020 model answering from memory alone, which is a low bar. The paper also names two problems it is trying to fix that had nothing to do with lying: providing provenance for a decision, and updating a model's world knowledge without retraining it.

So the honest description of RAG is that it makes a model's answers more current, more auditable and more grounded. Everything after that is marketing that got ahead of the citation.

The pipeline, and how much of it happens before the model wakes up

Here is what actually runs when someone types a question into a RAG-backed chatbot. It matters that you can see the whole thing at once, because the interesting failures are not where people look for them.

In January 2024 Scott Barnett and four co-authors at Deakin University published a catalog of seven failure points from three production RAG systems, including a biomedical case study with 4,017 documents and 1,000 questions.^[5] Their list: missing content, the answer never ranking in the top k, the passage getting dropped during consolidation, the model failing to extract an answer that is present, wrong format, wrong specificity, and incomplete answers. Count them. Three are pure retrieval failures. The fourth, the model not pulling out an answer that is sitting right there, the paper attributes to noise and contradictions in the passages retrieval handed over. Only the last three are about how the model was asked.

Failure one: the chunk boundary cuts the answer in half

Chunking is where most pipelines quietly break, and it breaks for a reason that sounds trivial until you see it in production. A chunk is embedded on its own, with no memory of the document it came from.

Anthropic's engineering team gave the cleanest example of this I have seen. Take a chunk from an SEC filing that reads “the company's revenue grew by 3% over the previous quarter.”^[3] Which company? Which quarter? The chunk does not say, because the sentence three paragraphs up said it. Embed that chunk and you get a vector that is a decent match for every revenue sentence in every filing you own and a great match for none of them.

The fix is unglamorous: prepend a short generated summary of the surrounding document to each chunk before embedding it. Anthropic measured a baseline top-20 retrieval failure rate of 5.7%, meaning that in 5.7% of queries none of the twenty retrieved chunks contained the answer. Contextual embeddings alone cut that to 3.7%. Adding keyword search took it to 2.9%. Adding a reranker took it to 1.9%.^[3]

Failure two: the search returns things that are similar, not things that are right

This is the one that took me longest to accept, because it feels like it should be a tuning problem and it is not. It is a capacity limit with a proof attached.

In August 2025, four researchers at Google DeepMind connected embedding retrieval to classic results in communication complexity and sign-rank, and showed that for any fixed embedding dimension there are combinations of documents that no single-vector model can ever return as a top-k result, no matter how well trained it is.^[4] Then they built LIMIT, a benchmark of short one-sentence documents with a thousand queries, deliberately simple. State-of-the-art embedders including Gemini Embeddings, Qwen3 and E5-Mistral struggle to reach 20% recall@100 on it.^[4]

BM25, a keyword ranking function first published in the 1990s and shipped in every serious search engine since, comes close to a perfect score on the same task.^[4]

The practical version of this is the failure every RAG team has seen. A user asks about error code 4032 and the retriever hands back three passages about error handling. It asks about the 2024 amendment and gets the 2019 original, because the two documents are 95% identical text and the vector cannot see the year. Dense embeddings blur exactly the tokens that carry the fact: identifiers, dates, part numbers, negations. Hybrid search exists because of this, not because someone wanted a more complicated architecture.

Failure three: the model reads the context and ignores it anyway

Say the retrieval worked. The right passage is sitting in the prompt. You are still not safe, and the reason cuts both ways.

Kevin Wu, Eric Wu and James Zou at Stanford built ClashEval to measure what happens when retrieved content and a model's internal knowledge disagree. They ran over 1,200 questions across six domains through six frontier models, deliberately corrupting the retrieved passages. Models overrode their own correct prior more than 60% of the time.^[6] Feed a model a wrong number in a retrieved chunk and the majority of the time it will repeat the wrong number, even when it knew better unprompted.

So RAG does not just fail to stop hallucination. It creates a new channel for it: a bad passage now becomes a confident, cited, wrong answer. Which is also, incidentally, why retrieved content is an untrusted input and belongs in your threat model rather than your happy path, something I got into in the piece on the prompt injection attack surface.

The failure runs the other direction too. Nelson Liu and colleagues at Stanford, Berkeley and Samaya AI documented in 2023 that model accuracy follows a U-shaped curve across the context window: performance is highest when the relevant text sits at the very beginning or the very end, and degrades in the middle.^[7] Retrieve twenty chunks, put the right one at position eleven, and you have hidden it in the one place the model reads worst. Ordering is not cosmetic.

Failure four: the citation is decoration

The footnote under a RAG answer looks like proof. It usually is not.

In most implementations the model is asked to produce citations as text, in the same generation pass as the answer, from the same probability distribution. Nothing checks afterwards that sentence four is actually supported by source two. Tianyu Gao and colleagues at Princeton built ALCE to measure precisely this, and found that on the ELI5 dataset even the best systems lacked complete citation support 50% of the time.^[8]

Half. The citation is generated, not verified, and a generated citation has exactly the reliability of any other generated token. If you want grounding you have to build it: verify each claim against the span it points at with a separate check, and refuse to display an answer whose support fails. That is engineering work nobody puts in the demo.

The thing I wish someone had told me two years ago

Retrieval quality is a hard ceiling on answer quality. Not an influence on it. A ceiling.

If the passage containing the answer is not in the context window, there is no prompt on earth that recovers it. The model cannot reason its way to a fact it was never shown. Every hour you spend rewording the system prompt when your recall@10 is 60% is an hour spent polishing the bottom four-fifths of a system whose top is missing. And most RAG teams spend their time exactly there, because prompts are easy to edit and retrieval evaluation is a project.

The operational consequence is that you need two metrics, not one. Track retrieval recall at your chosen k as its own number, on its own evaluation set, entirely separately from answer quality. When an answer is wrong, the first question is not “what should the prompt have said?” It is “was the answer in the context at all?” Those two branches lead to completely different work. Conflating them is how teams spend a quarter tuning a prompt to fix a chunking bug.

This is also why the retrieval layer is worth treating as real infrastructure rather than a library call, and why standardizing how models reach external data matters more than it sounds. It is the same reason context engineering has quietly replaced prompt engineering as the job title that describes the actual work.

What to actually do on Monday

Four things, in order, and the order is the point.

1. Build a retrieval evaluation set before anything else. Fifty real questions with the passage that answers each one labeled by hand. Measure recall@5, recall@10 and recall@20. You now have a number that moves.
2. Turn on hybrid search. Run BM25 alongside your vector search and fuse the results. On the LIMIT benchmark keyword search beat every frontier embedding model outright, and in Anthropic's numbers it took failure from 3.7% to 2.9%.^[3,4] It is the highest-value change per hour of work in this entire pipeline.
3. Add context to your chunks, then rerank. Prepend document-level context before embedding, retrieve wide, rerank with a cross-encoder, and pass the model a small number of well-ordered passages rather than a pile.^[3,9]
4. Verify citations after generation, not during it. Check each claim against the span it cites in a separate pass. If it fails, say so in the interface. A visibly abstaining system beats a confidently wrong one, and half of generated citations do not fully hold up.^[8]

RAG is a genuinely good technique. It made models current, auditable and cheaper to update, exactly as the 2020 paper said it would. What it never did was make them honest, and the industry sold the second thing while shipping the first. Fix your retriever. The model is fine.