Most RAG demos look great and most RAG products in production quietly hallucinate. The difference is almost never the language model. It is retrieval.
A demo gets built against a handful of clean documents, tested with a handful of friendly questions, and it works beautifully. Production is a different animal: thousands of documents with inconsistent formatting, users asking questions the documents do not directly answer, and a business that needs the system to be right, not just fluent.
Retrieval is the product
If the right passage never makes it into the context window, no amount of prompting will save the answer. The model will do its best with what it is given, and if what it is given is irrelevant or incomplete, a confident-sounding wrong answer is the likely output. We treat retrieval as the core system to engineer, and generation as the comparatively easy part that comes after.
This reframing changes where teams spend their time. Instead of iterating endlessly on prompt wording, the highest-leverage work is almost always upstream: how documents are chunked, how they are indexed, how candidates are ranked, and how the system decides what counts as relevant enough to use.
- Chunk on meaning, not on a fixed character count, so a passage carries a complete thought.
- Combine dense vector search with keyword search for hybrid recall that survives jargon and exact terms.
- Rerank candidates before they ever reach the model, so the best few passages lead.
- Cite sources on every answer, so a claim is auditable rather than merely plausible.
Chunking is a modeling decision, not a formatting step
Splitting documents every 500 characters is the fastest way to break retrieval. A fixed-size chunk cuts a table in half, separates a heading from the paragraph it introduces, or splits a step-by-step procedure across two vectors that no longer resemble either step. The retriever ends up choosing between two incomplete fragments instead of one complete answer.
Chunking on structure instead, respecting headings, list boundaries, and table boundaries, produces passages that actually stand alone. A chunk that is a complete thought is a chunk the model can use without guessing at what was cut off before or after it.
Hybrid search matters more than most teams expect. Pure vector search misses exact identifiers, error codes, and product names, because embeddings are built for semantic similarity, not exact string matching. Pure keyword search misses paraphrase, because a user who asks about the problem in their own words will not match the document's exact vocabulary. Running both and merging the results recovers the cases each one drops on its own.
Reranking is where accuracy actually gets won
Initial retrieval, whether vector, keyword, or hybrid, is optimized for recall: cast a wide enough net that the right passage is somewhere in the results. It is not optimized for precision. A reranking pass, scoring each candidate against the specific question rather than against a general similarity metric, is what turns a list of maybe-relevant passages into a short list the model can actually trust.
The gap between top-1 relevance before and after reranking is often the single biggest accuracy lever in the whole system, larger than any prompt tweak and cheaper than fine-tuning.
A grounded wrong answer is worse than no answer. Guardrails and citations are not polish, they are the safety layer.
Guardrails for the honest "I don't know"
A trustworthy system has to be willing to decline. When retrieval returns nothing relevant, the right behavior is to say so, not to improvise. We wire that in explicitly: a relevance threshold below which the system answers that it lacks the information, rather than filling the gap with a confident guess dressed up in the same fluent tone as a correct answer.
This is a harder design problem than it sounds. Set the threshold too aggressively and the system declines to answer questions it could actually handle, which frustrates users and undermines trust in a different direction. Set it too loosely and hallucinations creep back in. Getting the threshold right takes real questions and real judgment, not a default value copied from someone else's system.
Measure it or it will drift
We build an evaluation set from real user questions and score both retrieval quality and answer quality on every change. Without evals, a RAG system degrades silently the moment your data, your chunker, or your prompts change, and you find out from a user instead of a dashboard, usually at the worst possible time.
The eval set is not a one-time artifact. Every question a user asks that the system gets wrong becomes a new test case, and the suite becomes the definition of done for any future change. A system that passes its evals today and has no eval suite tomorrow is not actually more reliable, it is just less observed.
Treat the eval set the way you would treat a test suite for any other piece of production software: it grows with the system, it runs on every change, and a regression in it blocks a release. That discipline is the entire difference between a RAG system that stays accurate for a year and one that quietly gets worse without anyone noticing until a customer does.
Freshness is a retrieval problem too
Documents change. Policies get updated, pricing changes, product specs get revised, and a RAG system indexed once and left alone will keep confidently serving the old version long after it stopped being true. This is a quieter failure mode than an obvious hallucination, because the answer is fluent, well cited, and simply wrong in a way that is hard to catch without checking the source directly.
Building a re-indexing pipeline that picks up document changes on a real cadence, rather than treating the index as a one-time load, is not a nice-to-have for anything that touches operational or customer-facing content. The index has to be treated as a living reflection of the source material, not a snapshot.
Where teams get the architecture wrong
The most common mistake is treating retrieval as a solved problem you can drop in with a hosted vector database and a default embedding model, then spend all remaining effort on the prompt. That ordering is backwards. A mediocre prompt over excellent retrieval will usually outperform an excellent prompt over mediocre retrieval, because the model cannot answer accurately from context that never had the right information in it.
The second most common mistake is skipping citations because they add friction to the response format. Citations are not a UI nicety, they are what lets a user or a reviewer actually verify a claim instead of trusting it blindly, and they are what makes it possible to audit a wrong answer back to the specific passage that misled the system.
Get the retrieval architecture right first. Everything downstream of it, including the parts users actually see, gets easier once the system is working from the right information in the first place.