22 August 2026
RAG in production: the 7 mistakes that break quality (and how to fix them)
The 7 RAG mistakes I fix in production: chunking, retrieval evals, reranking, metadata filters, hallucination guardrails, latency, cost, and monitoring.
RAG quality almost never breaks all at once
A RAG system rarely looks bad on day one. The demo answers quickly, cites two documents, and creates enough confidence to launch a pilot. The real problems appear later, when actual users ask ambiguous questions, when the corpus grows, when documents contradict each other, or when an approximate answer starts influencing a business decision. At that point, quality is not mainly about picking a better model. It is about engineering the whole system around retrieval, evidence, product constraints, and feedback.
In production projects I see, especially around media and knowledge-heavy teams such as Brut, Les Echos, and quieter product organizations, the same failure modes keep coming back. They are not exotic. They are practical: naive chunking, no retrieval evaluation, stale documents in the index, no permission filtering, and prompts that ask the model to sound confident even when the sources are weak. The good news is that these problems are fixable when you treat RAG as a living product, not as a black box connected to a document folder.
Mistake 1 — Chunking for the engine, not for the job
The default move is to split every document into fixed-size chunks: 800 tokens, 100-token overlap, index everything, done. That can work for a simple demo, but quality drops as soon as the useful information depends on a heading, a definition, a table, an exception, or the paragraph just before the one retrieved. The chunk may be technically clean and semantically poor. The model receives a plausible excerpt, but not the actual unit of meaning needed to answer well.
The fix I apply is to start from the real document structure. I chunk by sections, headings, semantic blocks, pages, or business entities when possible. I attach useful metadata to each chunk: source, date, document type, confidence level, owner, version. Then I test several chunking strategies against real questions instead of choosing by intuition. Good chunking should make precise answers possible, but it should also make citations easy to verify. If the user cannot inspect the source, the RAG system is not ready yet.
Mistake 2 — Believing embeddings are enough
Vector search is powerful, but it is not magic. It connects nearby formulations; it does not automatically understand business truth. A question may include an internal acronym, a product name, a customer reference, a time period, or an exact term that matters more than broad semantic similarity. I have seen systems retrieve a document that feels extremely close in tone and is completely wrong in context. That is risky because the final answer still sounds smart.
In production, I often combine several signals: vector search, lexical search, controlled synonyms, business filters, then reranking over the best candidates. Reranking is especially useful when the corpus contains many similar documents. It helps move the right source above the merely plausible one, instead of letting the embedding model decide alone. I prefer a slightly more explicit, observable, and testable pipeline to an elegant search layer that fails silently on important cases.
Mistake 3 — Never evaluating retrieval
Many teams evaluate only the final answer. They read ten outputs, decide the system looks fine, and move on. The problem is that the generator can produce a good answer from the wrong sources, or a bad answer even when the right sources were retrieved. If you do not evaluate retrieval separately, you do not know where the system is failing. The team debates the model while the real issue may simply be that the correct document never reaches the top five results.
My fix is intentionally pragmatic: create a small set of 50 to 100 real questions with expected sources, acceptable answers, expected refusals, and trap cases. Measure recall, the rank of the right source, no-result cases, and failure patterns. Replay that set after every meaningful change to chunking, embeddings, filters, or reranking. Evals are not an academic layer. They prevent you from breaking what worked yesterday while improving what looked weak today.
Mistake 4 — Forgetting metadata, dates, and permissions
A RAG system that retrieves the right paragraph in the wrong context is still a bad RAG system. The classic examples are easy to recognize: a 2023 HR policy outranks the 2026 version, a sales deck appears before validated documentation, or a user receives information that exists in the index but should not be visible to them. The retrieval engine may look strong, but it breaks trust because it ignores content governance.
I filter before generation, not after the fact. Chunks need to carry the metadata that matters: validity date, language, owning team, confidentiality, customer scope, draft or approved status. Permissions must be applied at retrieval time, with dedicated tests. In a serious system, an old or unauthorized source should not be retrieved and then hidden by the prompt. It should never enter the model context in the first place.
Mistake 5 — Asking the model to guess the truth
When retrieval returns weak, contradictory, or incomplete sources, the model tends to smooth over the problem. It writes a confident answer, merges two documents, or invents the missing step to satisfy the user. That is not a moral failure. It is the natural behavior of a generator optimized to produce something useful. In RAG, the important question is not only: what does the model know? The better question is: what can it prove with the sources available right now?
I build simple and visible guardrails. The prompt should require source-grounded answers, short citations, and a clear refusal when the documents are insufficient. The interface should show the references used, not hide them in logs. For sensitive workflows, I sometimes add a verification step: does every important claim map to an excerpt? If not, the answer is rewritten or blocked. A good RAG experience can say: I do not have enough reliable evidence.
Mistake 6 — Optimizing quality while ignoring latency and cost
You can almost always improve an answer by adding more chunks, a larger model, a heavier reranker, a second critique pass, and a final synthesis step. The problem is that the user is waiting, and the bill grows with every request. A perfect pipeline at 18 seconds is not necessarily a good product. An excellent but expensive pipeline will be limited, bypassed, or turned off as usage increases.
I treat latency and cost as design constraints. I keep top-k reasonable, rerank only when it materially improves difficult cases, cache what can be cached, stream the first useful output, and choose models by step instead of by ego. A simple extraction does not need the same model as a risky synthesis. The right RAG system delivers enough quality for the decision, fast enough for the workflow, at a cost the business can afford every day.
Mistake 7 — Monitoring nothing after launch
RAG quality drifts. The corpus changes, users learn to ask new questions, documents get old, prompts are edited, and a useful source can disappear without making noise. If nobody looks at traces, the system can lose 20% of its quality before the team understands why users are going back to Slack or Google Drive. Launch is not the end of the work. It is the beginning of the maintenance loop.
I always want a few simple signals: user question, retrieved documents, retrieval scores, final answer, latency, cost, user feedback, fallback reason, and prompt version. Then I set a lightweight review habit: which question types fail, which sources are missing, which refusals are useful, which answers cost too much? This loop turns RAG into an improvable system. Without monitoring, you hope. With monitoring, you operate.
Good RAG is a product that learns
Most RAG quality problems do not require a more complicated architecture. They require a more honest one: clean documents, measured retrieval, trust rules, an interface that exposes sources, and a team that keeps watching what happens after production launch. That is less spectacular than a magical demo, but much more valuable.
When I take over an existing RAG system, I rarely start by changing the model. I inspect real questions, retrieved sources, user-facing failures, and product constraints. Then I fix things in order: chunking, retrieval, filters, evals, guardrails, latency, monitoring. That discipline is what moves a document assistant from impressive in a meeting to genuinely useful in daily work.