Building a RAG Pipeline That Survives Production

We have now shipped eleven retrieval-augmented generation systems into production. The first three were harder than they should have been, and for the same reason every time: we treated retrieval as solved and spent our attention on the model. That is backwards.

A RAG demo built in an afternoon will answer your ten favourite questions beautifully. A RAG product has to answer the ten thousand questions you did not think of, over documents that change weekly, for users who will not forgive a confident wrong answer. Here is what actually separates the two.

Chunking is a retrieval decision, not a preprocessing chore

Fixed 512-token chunks are the default in every tutorial and the wrong answer for most real corpora. A policy document has structure — sections, clauses, tables — and splitting on token count shreds it. The chunk that contains the answer ends up missing the heading that gives it meaning.

What worked consistently for us:

• Split on document structure first (headings, clauses, list boundaries), then subdivide only oversized sections.
• Prepend the heading path to every chunk, so "Clause 4.2" carries "Leave Policy > Sick Leave > Clause 4.2" into the embedding.
• Overlap by one sentence, not by a fixed token count, so you never cut mid-thought.
• Keep tables intact and store them as markdown. Splitting a table across chunks destroys it.

Pure vector search is not enough

Semantic search is excellent at paraphrase and terrible at exact identifiers. Ask a vector index for "policy HR-2019-14" and it will cheerfully return five documents about HR policies from 2019 and not the one you named.

Hybrid retrieval fixes this: run BM25 keyword search and vector search in parallel, then fuse the result sets. In our benchmarks, hybrid retrieval improved recall by 18 to 31 percentage points over vectors alone, with the biggest gains exactly where it matters — queries containing codes, names and dates.

Rerank before you generate

Retrieval gives you twenty plausible passages. The model only needs the best four, and stuffing all twenty into context makes answers worse, not better — the relevant passage gets buried and latency and cost both climb.

A cross-encoder reranker scores each passage against the actual query rather than comparing pre-computed embeddings. It costs one extra call and it was, in every project we measured, the single highest-return component we added.

Build the evaluation set before you build the system

If you cannot say how accurate your RAG system is, you do not have a RAG system. You have a demo with good lighting.

Sit with the people who will use it and collect 200 to 500 real questions with verified correct answers and their source documents. This is unglamorous work and it is the difference between shipping with confidence and shipping with hope.

Measure three things separately, because they fail separately: retrieval recall (did the right passage get fetched at all), answer faithfulness (is the response actually supported by the fetched passages), and citation accuracy (does the cited source really say that). A system can score well on the last two while quietly failing the first.

Make refusal a first-class outcome

The most valuable behaviour in an enterprise RAG system is a clean "I could not find this in the available documents." Users forgive that instantly. They do not forgive a fluent, cited, wrong answer — and one of those can cost you the entire deployment.

Enforce it structurally: require supporting passages above a relevance threshold before the model is allowed to answer, and surface citations inline so any user can verify a claim in two seconds. Trust is built by being verifiable, not by being confident.

What we would tell our past selves

1. Spend 70% of your effort on retrieval quality and 30% on prompting. We had it reversed for a year.
2. Instrument everything — log the query, the retrieved passages, their scores and the final answer. You cannot debug what you did not record.
3. Re-index on a schedule and verify it. Stale knowledge is the most common production failure and the quietest.
4. Enforce document permissions at retrieval time, never at display time.