What Is RAG? Grounding AI in Your Own Documents

RAG — Retrieval-Augmented Generation — retrieves relevant passages from your own documents and inserts them into the prompt before the model answers. It is the standard answer to "make the AI know about our company."

The pipeline

Offline, once per document:

1. Split the document into chunks.
2. Convert each chunk into an embedding — a vector representing its meaning.
3. Store the vectors in a vector database.

At query time, per question:

1. Embed the question.
2. Find the most similar chunks.
3. Insert them into the prompt.
4. Ask the model to answer using only that context.

The model itself is unchanged. You are not teaching it anything; you are handing it the right reference material at the right moment.

Why not just fine-tune?

This is the most common architectural question, and the answer is fairly settled.

Fine-tuning teaches behaviour — style, format, tone, classification conventions. It adjusts how the model responds.

RAG provides knowledge — facts, documents, current data. It changes what the model has access to.

Fine-tuning to add facts works poorly. The model may absorb the phrasing of your documents without reliably learning their content, and you get confident errors that are hard to trace. It is also expensive to update: new information means retraining.

RAG updates instantly when a document changes, can cite its sources, and costs nothing to maintain beyond the pipeline. For knowledge, it wins on every axis that matters.

Why not just paste everything?

With million-token context windows, why retrieve at all?

Three reasons. Cost — you pay for every token on every request. Attention — models under-weight the middle of long contexts, so a fact buried in 500,000 tokens may effectively be invisible. And scale — most real document collections exceed even the largest window.

Retrieval is a precision tool. Passing three relevant paragraphs beats passing three hundred pages.

The prompt that makes it work

The single most important line in any RAG system:

Answer using ONLY the context below. If the context does not
contain the answer, say "Not found in the provided documents."
Do not use outside knowledge. Cite the document name for each claim.

Context: {retrieved chunks}
Question: {user question}

Without the "only from context, otherwise say not found" instruction, the model blends retrieved material with its parametric memory. You get answers that look sourced but are not — the most dangerous failure mode available, because it defeats the audit trail you built RAG to get.

Chunking is where quality is won or lost

Chunk size is the most consequential tuning decision.

Too large and relevance dilutes — a 3,000-token chunk about ten topics matches weakly on all of them. Too small and context is lost — a sentence pulled from its section may be unintelligible.

Reasonable defaults: 300-800 tokens per chunk, 10-20% overlap, split on natural boundaries like headings and paragraphs rather than fixed character counts. Overlap prevents a relevant passage being cut across a boundary.

The highest-leverage refinement: prepend a context header to every chunk.

From: 2026 Refund Policy > International Orders

[chunk text]

Retrieval accuracy on ambiguous queries improves markedly, because the embedding now carries the passage's place in the whole document rather than just its local content.

Retrieval quality beats model quality

When RAG answers are bad, the instinct is to upgrade the generation model. Usually the actual problem is upstream: the right passage was never retrieved, or was retrieved at rank 12 and never made it into the prompt.

Two fixes with outsized impact:

Hybrid search. Run both keyword (BM25) and semantic search, then merge. Semantic search matches meaning but can miss exact terms — product codes, error numbers, specific names. Keyword search catches those. Together they outperform either alone.

Reranking. Retrieve 50 candidates cheaply, then rerank with a cross-encoder that examines the query and each candidate together. Cross-encoders are far more accurate than embedding similarity and affordable over 50 items.

Document quality matters too

Retrieval quality starts before any code runs. Documents written as clear sections with descriptive headings, one topic each, retrieve far better than long unstructured prose.

The specific thing to eliminate is cross-references. "As mentioned above" breaks completely when a chunk is retrieved in isolation. Every section should make sense read alone, because that is how retrieval will present it.

Rewriting your ten most-queried documents for retrieval often improves answers more than any pipeline change.

When RAG is the wrong tool

RAG suits question answering over a document collection. It suits less well:

• Aggregation across many documents — "how many contracts mention X" is a database query, not a retrieval problem.
• Whole-document reasoning — summarising a contract as a whole needs the whole contract, not three chunks.
• Highly structured data — if it lives in tables, query the tables.

Getting started

For a small collection, you may not need infrastructure at all. Custom GPTs and Claude Projects give you document upload with retrieval built in, no code required. That is the right first step — validate that retrieval solves your problem before building a pipeline.

Our RAG & Custom AI Knowledge course covers chunking, embeddings, retrieval quality and the no-code path in depth.