A retrieval-augmented generation demo can look convincing with a handful of clean documents and a carefully worded question. Production is where the uncomfortable questions arrive: Which source was current? Why did this result rank first? What happens when retrieval returns nothing? Can we reproduce the answer?

The useful shift is to stop treating RAG as one model call. It is a chain of decisions with observable inputs and outputs.

Start with the retrieval contract

Before choosing an embedding model or vector database, define what the system must retrieve. Identify the unit of knowledge, the freshness requirement, the access rules, and what a grounded answer must cite.

A practical retrieval contract includes:

  • the source types and their owners;
  • the acceptable delay between a source change and index availability;
  • the metadata required for filtering and authorization;
  • the fallback when evidence is weak or missing;
  • the signals you will retain for evaluation.

This contract turns vague quality complaints into engineering decisions.

Keep ingestion reproducible

An ingestion pipeline should make it possible to answer a simple question: how did this chunk get into this index?

Store source identifiers, timestamps, parser and chunking versions, and the embedding configuration. Make updates idempotent. Separate document access rules from prompt instructions; authorization belongs in retrieval, not in a sentence asking the model to behave.

Measure the stages separately

End-to-end answer quality matters, but it does not tell you where a failure happened. Keep stage-level signals for query transformation, candidate retrieval, ranking, context assembly, generation, and citation.

Start with a small evaluation set drawn from real user tasks. Include difficult negatives and questions the system should decline. Review failures by category instead of compressing every issue into one score.

Design for weak evidence

The system needs an honest state for “I do not have enough evidence.” Thresholds alone are rarely sufficient, but a combination of retrieval signals, source coverage, and answer-level checks can create a useful guardrail.

The product experience matters here. A clear partial answer with visible sources is often more useful than a confident paragraph that hides uncertainty.

Observe the whole path

In production, retain the identifiers needed to trace a request without logging sensitive content indiscriminately. Useful events include retrieval latency, result counts, filters applied, source freshness, model and prompt versions, token usage, and user feedback.

RAG becomes maintainable when the team can connect a poor answer to a specific stage, reproduce it, and test the change that should fix it.

Separate retrieval metrics from answer metrics

For a labeled query set, retrieval can be evaluated before generation:

  • Recall@k: did the top k contain at least one answer-bearing passage?
  • Mean reciprocal rank: how early did the first relevant passage appear?
  • nDCG: when relevance has grades, did the ranking place higher-value evidence first?

These metrics still depend on judgments about relevance. Record the corpus/index version and the exact query transformation, filter, and reranker configuration. Otherwise a score cannot be reproduced.

For the generated response, assess claim-level support: extract factual claims, map each to cited evidence, and distinguish unsupported claims from incomplete answers. A fluent answer with perfect retrieval can still ignore or contradict the context.

The production test

A production RAG system is not finished when it produces a good answer. It is ready when the team can explain where the answer came from, enforce who may see it, notice when quality changes, and improve the system without guessing.

References