Retrieval you can trust

Retrieval-augmented generation sounds like a model technique. In production it is an information-access system that happens to end in a model. Whether its answers can be trusted is decided upstream: by which documents went in, how they were split, who is allowed to see them, and whether they are still current.

What production retrieval really is#

The basic idea, from a 2020 paper by Lewis and colleagues, is simple: before answering, look up relevant passages and give them to the model. Enterprises like it because their knowledge changes faster than any model is trained, is private, and has to be cited.

A production system adds everything the basic idea leaves out. It needs to know who is asking and what they may see, which version of a document is authoritative, whether the index is current, how to cite precisely, and what to do when the evidence is missing or contradictory. Get those right and an ordinary model gives excellent answers. Get them wrong and the best model in the world produces confident, well-written mistakes.

When not to use it#

Two pipelines#

A retrieval system is two systems that meet at the index. Ingestion runs in the background, turning source documents into searchable, permission-tagged chunks. The query path runs per request. Most quality problems start in the first and are noticed in the second.

Ingestion decides quality#

A system cannot retrieve what was never indexed correctly. Ingestion failures are quiet: a page that failed to parse, a table that lost its headers, a folder the connector lost access to. Build ingestion to be observable from day one.

• Parsing with quality checks. Record pages parsed, tables found and OCR confidence per document. A policy document where half the tables failed to extract should be marked failed, not indexed.
• Tables keep their headers. A row reading "Enterprise | 99 | Yes" means nothing without the column names. Store each row with its labels.
• Metadata on every chunk: source, owner, version, effective and expiry dates, sensitivity, and the permissions it inherits.
• Versions and deletion. Link chunks to document versions so superseded text can be removed. Deletion requests must reach the index too.
• Smoke tests after every run. A handful of known questions that must retrieve known documents. If they stop working, the run failed.
• A plan for changing the embedding model. Mixing vectors from two models quietly ruins search. Re-embed everything, or keep versioned indexes.

Chunking follows structure#

A chunk is the unit that gets retrieved, cited and billed as context, so chunking is a design decision, not a default. Fixed-size splits cut across headings, tables and clauses. Split along the document's own structure instead: sections and headings for policies, whole functions for code, rows with headers for tables, question and answer pairs for FAQs.

A pattern that works well for enterprise documents: retrieve small chunks for precision, then hand the model the larger section each one came from, so it has context. Whatever you choose, test it against real questions. There is no universally right chunk size, only the one your evals support.

The query path#

Identity and filters come first

Resolve who is asking, then filter the search to documents they may see, before ranking anything. Permissions are not a ranking signal or a prompt instruction. They are a server-side rule that no request can bypass. Add a check after retrieval too: if any chunk belongs to a tenant or permission group the user doesn't, block the response and alert. Multi-tenant leaks are the most serious incident a retrieval system can have.

Search two ways, then rerank a short list

Semantic search finds passages with similar meaning. Keyword search finds exact terms: product codes, clause numbers, error messages, names. Real questions need both. Merge the two result lists, then use a reranker on a short list to choose the best handful. Reranking is the slow, expensive step, so keep its input small.

Build the context, then answer or abstain

Remove duplicate chunks, order the rest by authority and relevance, and attach citations. Instruct the model to answer only from the context and to say when it can't, then check the answer: are the claims supported by the cited passages, and do the citations point to the right ones?

Treat retrieved text as data, never as instructions. A document can contain text such as "ignore previous instructions". Chapter 13 covers how to keep that from mattering.

Vector search returns candidates, not truth. Trust comes from filters, authority, citations and the willingness to say "I don't know".

Freshness and authority#

An answer can be perfectly grounded in the index and still be wrong, because the index is behind the source. This is the most common way retrieval systems fail in regulated settings. It is not hallucination, and it won't show up in a model eval.

• Track freshness per document (source last changed vs. last indexed), not just whether the connector is running. Connectors fail on single folders while reporting healthy overall.
• Agree a freshness target per source with its owner: an hour for incident runbooks, a day for policies, a week for archives.
• Show 'last updated' next to citations, so users can judge for themselves.
• When a critical source is past its target, warn the user or route those questions to a person rather than answer silently.
• Plant a test document in each important folder and check that it appears in the index. It catches permission changes and folder moves.