Why RAG Systems Fail in Production — and How to Engineer Them Properly

Retrieval-Augmented Generation looks deceptively simple: embed your documents, retrieve the top matches, stuff them into the prompt, and let the model answer. The demo works on three hand-picked questions. Then it meets real users and real documents, and accuracy collapses. The failure is almost never the LLM — it is the retrieval pipeline around it.

1. Chunking is a retrieval decision, not a preprocessing detail

Fixed 1,000-character chunks split sentences, separate a claim from its qualifier, and destroy table structure. Chunk on semantic boundaries — headings, paragraphs, list items — and keep a small overlap so context that straddles a boundary is not lost.

# Overlapping, boundary-aware chunking
def chunk(text, size=800, overlap=120):
    paras = [p for p in text.split("\n\n") if p.strip()]
    chunks, buf = [], ""
    for p in paras:
        if len(buf) + len(p) > size:
            chunks.append(buf.strip()); buf = buf[-overlap:]
        buf += "\n\n" + p
    if buf.strip(): chunks.append(buf.strip())
    return chunks

2. Pure vector search is not enough

Dense embeddings capture semantic similarity but miss exact terms — product codes, error numbers, names. Combine dense retrieval with sparse keyword search (BM25) in a hybrid retriever and fuse the results. This single change recovers the "obvious" matches that embeddings silently drop.

3. Add a reranker

Top-k by cosine similarity is a coarse filter. A cross-encoder reranker re-scores the candidate set against the actual query and consistently lifts precision@k. Retrieve broadly (k=20–50), rerank, then pass only the best 4–8 chunks to the model. Fewer, better chunks beat more, noisier ones.

4. Ground the generation and force citations

System: Answer ONLY from the context below. If the answer is not present,
say "I don't know." Cite the source id you used in square brackets.

Context:
[doc:14] ...retrieved chunk...
[doc:31] ...retrieved chunk...

Question: {user_question}

Grounding plus an explicit "I don't know" path is what separates a trustworthy assistant from a confident fabricator.

5. You cannot improve what you do not measure

Build an evaluation set of real questions with known answers. Track retrieval metrics (recall@k, MRR) separately from answer metrics (faithfulness, answer relevance). When quality drops, you need to know whether retrieval missed the chunk or the model ignored it — they have completely different fixes.

The takeaway

Production RAG is an engineering system with four tunable layers — chunking, hybrid retrieval, reranking, grounded generation — wrapped in continuous evaluation. Treat it that way and the "the model is hallucinating" complaints largely disappear.