Retrieval-augmented generation (RAG)

How the pipeline works

Two halves. Indexing, done ahead of time: split documents into chunks, turn each chunk into an embedding vector with an embedding model, and store the vectors with their text — commonly in Postgres with pgvector, or a dedicated vector database. Query time: embed the question, find the nearest chunks, paste the top few into the prompt with an instruction to answer only from them, and ask the model to cite which chunk it used.

That last part is not decoration. Citations are what make the answer auditable, and without them you cannot tell grounding from guessing.

Retrieval quality is where the work is. Pure vector search misses exact terms — product codes, error numbers, names — so most serious systems run hybrid search, combining vector similarity with keyword matching, then rerank the candidates with a cross-encoder before selecting. Chunking matters more than people expect: chunks that cut a table in half or separate a heading from its content produce retrieval that looks fine and answers that are subtly wrong.

Why it exists

It answers three problems at once: knowledge after the training cutoff, private data the model never saw, and hallucination on specifics. It also updates instantly — add a document and it is available on the next query, with no retraining.

What to watch out for

RAG does not teach the model anything, and it does not guarantee a grounded answer. Two separate misconceptions:

• It is not learning. Nothing persists between calls. Every question re-retrieves and re-sends. Compare with fine-tuning, which changes behaviour and style but is a poor way to install facts. Facts belong in retrieval; form belongs in fine-tuning.
• It is not a hallucination cure. If retrieval returns nothing relevant, most models will answer anyway from memory. You have to instruct and test for the "not in the provided documents" case explicitly, and check that it actually happens.

The failure mode nearly everyone hits: the answers are bad and the model gets blamed. In practice the retrieval step is at fault far more often. Before changing model, log what was retrieved for the failing questions. Usually the right chunk was never in the prompt at all.

Large context windows have not made this obsolete. Sending three relevant pages is cheaper, faster and more accurate than sending the whole manual and hoping.