RAG in production: what actually decides answer quality?

In a production RAG system, answer quality is decided mostly by retrieval: whether the right passages are found, ranked near the top and placed in the prompt in a usable form. The language model matters, but it cannot answer well from the wrong context. After retrieval, the deciding factors are how you parse and chunk your content, whether you combine keyword and vector search and rerank the results, and whether you measure all of this continuously against real questions, while enforcing permissions, keeping the index fresh and defending against injected instructions.

Retrieval-augmented generation was introduced by Lewis and colleagues in 2020 as a way to combine a model's built-in knowledge with an external, searchable document index. The idea is now the default architecture for assistants that answer from company documents, product catalogues, policies and support content. Building a demo takes an afternoon. Getting reliable answers from messy real documents, for real users with different permissions, is where the work is. This article collects what the research and the main vendors' documentation say about each stage, and where we think teams most often go wrong.

The pipeline, stage by stage

1. Ingest and parse: pull documents from their sources and turn PDFs, HTML, slides and spreadsheets into clean text with structure (headings, tables, lists) preserved.
2. Chunk and enrich: split documents into passages, and attach metadata such as title, section, date, source and access rights.
3. Embed and index: compute vector embeddings for each chunk and index the text for keyword search as well.
4. Retrieve: for each question, fetch candidates from both indexes, filtered by what the user is allowed to see.
5. Rerank: score the candidates against the question with a more precise model and keep the best.
6. Assemble context: order, deduplicate and trim the passages to fit the prompt.
7. Generate: ask the model to answer only from the supplied passages, with citations, and to say so when the passages do not contain the answer.
8. Guard and log: check inputs and outputs, record what was retrieved and generated, and feed that into evaluation.

Every stage can lose the answer. That is why improving the model alone so often disappoints: if the right passage never reaches the prompt, a better model just writes a more fluent wrong answer.

Where RAG systems fail

A useful map of failure comes from Barnett and colleagues, who drew on three case studies in research, education and biomedicine to describe seven failure points when engineering a RAG system. Two of their conclusions matter for anyone planning a project: validation is only really feasible once the system is in operation, and robustness is something that evolves over time rather than being designed in at the start. In other words, plan for continuous evaluation from day one.

Parsing and chunking: the unglamorous foundation

Most poor answers we see trace back to ingestion. A PDF table flattened into a stream of numbers, a header repeated on every page, a policy whose exceptions live in a footnote that was dropped: no retrieval technique downstream can recover what parsing threw away. Budget real time for parsing, especially for PDFs, scanned documents and spreadsheets, and inspect samples of the parsed output by eye before indexing anything.

Chunk size is a trade-off. Small chunks match questions precisely but lose context; large chunks keep context but dilute the embedding and waste prompt space. Microsoft's Azure AI Search documentation suggests starting with chunks of 512 tokens and an overlap of 25%, and adjusting from there. Microsoft's own retrieval experiments are consistent with that starting point: in vector-only tests, 512-token chunks reached a Recall@50 of 42.4, against 34.9 for chunks of 8,191 tokens, and a 25% overlap scored slightly better than no overlap. Those experiments did not test chunks smaller than 512 tokens, so treat the number as a default to measure against, not an optimum.

Contextual enrichment has some of the clearest published evidence. Anthropic reported that prepending a short, model-generated context of 50 to 100 tokens to each chunk reduced failed retrievals by 35%, measured as the share of relevant chunks missing from the top 20. Applying the same context to keyword search as well raised the reduction to 49%. It is a vendor's own benchmark, but the technique is cheap to test on your own data.

Embeddings: choose on your data, plan for change

The embedding model turns text into vectors so that similar meanings land close together. Public leaderboards are a useful shortlist, but they do not choose for you. The authors of the MTEB benchmark, which spans 8 task types, 58 datasets and 112 languages, found that no single embedding method dominated across all tasks. Your domain vocabulary, languages and question style decide which model works best, so compare two or three candidates on a set of your own real questions before committing.

• Languages. If users ask in Arabic about documents written in English, test cross-lingual retrieval explicitly; many models are much weaker across languages than within one.
• Dimensions. Larger vectors can capture more, but cost more to store and search. Some models let you request shorter vectors directly; OpenAI's documentation, for example, describes a dimensions parameter that shortens embeddings while keeping most of their usefulness.
• Input limits. Embedding models have a maximum input length, and text beyond it is truncated or rejected. Chunking has to respect it.
• Versioning. Vectors from different models are not comparable. Changing model means re-embedding the whole corpus, so store the model name and version with every vector and plan re-indexing as a routine operation.
• Domain terms. Product codes, legal references and internal acronyms are often poorly represented by general-purpose embeddings. That is the strongest argument for hybrid search.

Hybrid search: keywords still matter

Vector search finds passages that mean the same thing in different words. Keyword search, usually BM25, finds passages that contain the exact terms: an error code, a part number, a person's name, a clause reference. Real questions need both. The BEIR benchmark, which tested retrieval systems across 18 datasets without domain-specific training, concluded that BM25 is a robust baseline that dense retrievers often failed to beat outside the domain they were trained on.

Combining the two consistently helps. In Microsoft's tests on customer datasets, measured by NDCG@3, keyword search scored 40.6, vector search 43.8 and hybrid search 48.4. The same post notes the exception that proves the point: on queries that were themselves keyword-style, keyword search alone beat even hybrid search with reranking.

The usual way to merge the two result lists is reciprocal rank fusion, introduced by Cormack, Clarke and Büttcher in 2009. Each document's score is the sum of 1 / (k + rank) across the lists it appears in; Elasticsearch's implementation uses a default constant of 60 and notes that RRF requires no tuning. Because it uses ranks rather than raw scores, it avoids the awkward problem of comparing a BM25 score with a cosine similarity. You do not need a dedicated vector database to do this: Postgres with the pgvector extension can combine vector search with built-in full-text search and merge the results the same way.

Reranking: the highest-leverage step most prototypes skip

First-stage retrieval has to be fast over millions of chunks, so it compares a precomputed vector or index entry with the question. A reranker reads the question and each candidate passage together, which is slower but much more precise. The idea was established for neural search by Nogueira and Cho, whose BERT-based reranker improved on the previous state of the art by 27% in MRR@10 on the MS MARCO passage benchmark.

The production evidence points the same way. Adding Microsoft's semantic reranker to hybrid search raised NDCG@3 on its customer datasets from 48.4 to 60.1. In Anthropic's contextual retrieval experiments, adding a reranker on top of contextual embeddings and keyword search took the reduction in failed retrievals to 67%. The BEIR authors noted the price: rerankers achieved the best zero-shot results, but at high computational cost.

The practical pattern is to retrieve generously and rerank narrowly: fetch a few dozen to a hundred or so candidates from hybrid search, rerank them, and pass the best handful to the model. Hosted rerankers, such as Cohere's Rerank, and open-source cross-encoders both fit this slot. Measure the latency the reranker adds against the quality it buys, and cap the number of candidates accordingly.

If you only have time to improve one thing in a RAG prototype, add keyword search and a reranker before you touch the model. It is usually the cheapest quality you will ever buy.

Context assembly: what the model actually sees

Retrieval produces a ranked list; the prompt is a page of text. How you turn one into the other matters. Liu and colleagues showed that language models are often best at using information at the beginning or end of a long context, and worse when the relevant passage is buried in the middle. Stuffing the prompt with every candidate is therefore not a safe default. Put the strongest passages first, remove near-duplicates, keep each passage's title and source so the model can cite it, and stop when additional passages add more noise than information.

Long-context models change the trade-off for small collections. Anthropic's guidance is that if a knowledge base is smaller than about 200,000 tokens, it can be simpler to put all of it in the prompt and use prompt caching than to build a retrieval pipeline at all. For larger, changing or permissioned collections, retrieval remains necessary, but it is worth asking the question before building one.

Evaluation: the part that decides whether it keeps working

Every improvement above is a guess until it is measured on your own questions. The most valuable asset in a RAG project is an evaluation set: a few hundred real or realistic questions, each with the passages that should be retrieved and, where possible, a reference answer. Collect them from support tickets, search logs, subject-matter experts and early users, and keep adding the questions the system gets wrong.

Measuring retrieval and generation separately is what makes failures diagnosable. If faithfulness is high but correctness is low, the model is faithfully repeating the wrong passages: fix retrieval. If retrieval recall is high but faithfulness is low, the model is wandering from its sources: fix the prompt, the context assembly or the model. Frameworks such as RAGAS implement reference-free versions of several of these metrics, covering retrieval focus, faithfulness and answer quality.

Most answer-quality metrics at scale rely on a language model as the judge. That is reasonable with care. Zheng and colleagues found that strong models used as judges reached over 80% agreement with human preferences, about the same level at which humans agree with each other, but also documented position, verbosity and self-enhancement biases. Calibrate any automated judge against a sample of human-labelled answers, re-check it when you change models, and never let the system being tested grade itself without that check.

• Offline: run the evaluation set on every change to chunking, embeddings, retrieval, prompts or models, and block releases that regress.
• Online: log the question, the retrieved passages, the answer and the user's reaction (thumbs, follow-up questions, escalations to a human), with appropriate privacy controls.
• Review: read a sample of real conversations every week. Metrics show where to look; reading shows why.

Freshness: keeping the index honest

A RAG system that answers from last quarter's price list or a withdrawn policy is worse than one that says it does not know. Freshness is a data-engineering problem more than an AI one, and it needs the same discipline as any other pipeline.

• Incremental sync: detect new, changed and deleted documents at the source and update only those. Deletions are the step most often forgotten, and stale chunks keep being retrieved long after the source is gone.
• Dates as metadata: store effective and updated dates with each chunk, let retrieval prefer recent versions, and show dates in citations.
• Versioned documents: when several versions of a policy exist, index the current one, or label versions clearly so the model can tell them apart.
• Index lag as a metric: measure the time from a change at the source to its appearance in answers, and alert when it grows.
• Cache invalidation: if you cache answers, expire them when their source documents change.

Permissions: filter before retrieval, never after

In a company assistant, different people may see different documents. The only safe place to enforce that is in retrieval itself, so that passages a user cannot see never reach the prompt. Filtering the answer afterwards does not work: the model may already have summarised or paraphrased restricted content. OWASP's Top 10 for LLM applications now includes vector and embedding weaknesses as a category of its own, covering unauthorised access, leaks between tenants sharing one vector store and poisoned content in the knowledge base. Its recommended mitigations are permission-aware retrieval, partitioning data by user group and validating sources.

The common implementation is to copy each document's access-control list into the index as metadata and filter every query by the caller's identity and groups. Microsoft documents this as a security filter pattern for Azure AI Search. Two details catch teams out. First, permissions change: the access list in the index has to be synced as carefully as the content. Second, approximate vector indexes can interact badly with filters. pgvector's documentation, for example, notes that with its default settings a filter matching a small share of rows can return far fewer results than requested, and describes iterative index scans as the fix. Test retrieval quality for users with narrow permissions, not just for administrators.

Guardrails: retrieved text is untrusted input

Prompt injection is first on OWASP's list of LLM application risks, and RAG makes it worse. Greshake and colleagues showed that attackers can plant instructions in content an application is likely to retrieve, so that the model follows them as if they came from the user. Any document a third party can influence, from web pages to inbound emails to uploaded CVs, is a possible attack path.

• Treat retrieved passages as data, clearly delimited from instructions in the prompt, and tell the model not to follow instructions found inside them. This reduces the risk; it does not remove it.
• Give the model as little power as possible. An assistant that can only answer questions is far less dangerous to inject than one that can send emails or update records. Require human confirmation for consequential actions.
• Validate and sanitise model output before it is rendered or passed to other systems, as you would any untrusted input.
• Require citations and check that they exist and support the claim; refuse when the evidence is missing.
• Control what gets indexed: known sources, authenticated uploads, and scanning for hidden text in documents.
• Log enough to investigate incidents, and red-team the system with injected documents before launch.

Our guide to cybersecurity services covers the wider security review an AI feature should go through before launch.

Cost and latency

In most RAG systems, generation dominates both latency and running cost; embedding the question and searching the index are comparatively quick. Reranking adds a noticeable step that grows with the number of candidates. Ingestion is a one-off cost per document version, which becomes significant when you enrich every chunk with a model or re-embed the corpus after changing models.

The best cost optimisation is usually better retrieval. When the top five passages are the right ones, you can send five instead of twenty, which shortens the prompt, speeds up the answer and often improves it. Streaming the answer as it is generated makes the wait feel shorter even when it is not.

A practical order of work

1. Collect a first evaluation set of real questions with known answers before building anything elaborate.
2. Get parsing right for your document types, and inspect the output by eye.
3. Start with structure-aware chunks of a few hundred tokens, with titles and section headings attached.
4. Use hybrid retrieval from the start, merged with reciprocal rank fusion, and filter by permissions in the query.
5. Add a reranker and measure retrieval recall and nDCG before and after.
6. Tune context assembly: fewer, better passages, strongest first, with sources for citation.
7. Write the generation prompt to answer only from the passages, cite them and decline when they are insufficient.
8. Add guardrails, logging and an evaluation gate in the release pipeline.
9. Only then compare language models, using the same evaluation set.

For the wider question of where AI belongs in a product, and when it does not, see our guides to adding AI to an existing product and what AI development services involve.

How Authect approaches RAG projects

We treat a RAG system as a search and data-engineering project with a language model at the end, not the other way round. Our work usually starts with your documents and your users' real questions, an evaluation set built from them, and a measured baseline, so that every later change can be judged on evidence. Every project is led by a senior engineer, we use AI to speed up delivery with an engineer reviewing every line, and you own the code, infrastructure, indexes and evaluation data from day one.

Our generative AI and LLMOps work covers the full path from prototype to a monitored production system. If you have a RAG prototype that answers well in demos and poorly in real use, talk to the founders and we will help you find where the answers are being lost.