RAG & Context

RAG & Custom AI Knowledge

Ground AI answers in your own documents with Retrieval-Augmented Generation — from no-code Custom GPTs and Claude Projects to understanding chunking, embeddings, and retrieval quality.

What Is RAG? Grounding AI in Your Own Documents

Retrieval-Augmented Generation lets a general model answer questions about your specific data without retraining. Here is how the pipeline works and the details that determine whether it works well.

RAG (Retrieval-Augmented Generation)

RAG retrieves relevant passages from your own documents and inserts them into the prompt before the model answers. It grounds responses in your data, cuts hallucination, and needs no retraining.

Embedding

An embedding is a list of numbers representing the meaning of a piece of text, such that semantically similar texts have mathematically similar vectors. Embeddings make meaning-based search possible.

Vector database

A vector database stores embeddings and finds the most similar ones to a query vector quickly. It is the retrieval layer of most RAG systems. Examples include Pinecone, Weaviate, Qdrant and pgvector.

Chunking

Chunking splits documents into smaller passages before embedding them for retrieval. Chunk size is a key quality lever: too large dilutes relevance, too small loses the context needed to make sense.

Semantic search

Semantic search finds results by meaning rather than exact keywords, using embeddings to compare concepts. It matches "how do I get my money back" to a document titled "Refund Policy".

Reranking

Reranking takes an initial set of retrieved candidates and reorders them with a more accurate but slower model. It is one of the cheapest ways to materially improve RAG quality.

Knowledge cutoff

A model's knowledge cutoff is the date after which it has no training data. It cannot know about events, releases or prices after that point unless given the information in the prompt or via search.

Grounding

Grounding means tying model output to verifiable sources — retrieved documents, search results, or database records — rather than relying on parametric memory. Grounded answers can be checked.

Inference cost

Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.

Confidence score

A confidence score estimates how reliable an output is. Language models do not natively produce calibrated confidence, and asking a model how sure it is yields poorly calibrated self-assessment.

Cosine similarity

Cosine similarity measures how closely two vectors point in the same direction, on a scale from -1 to 1. It is the standard way to compare embeddings, because it captures semantic similarity while ignoring text length.

Context engineering

Context engineering is the practice of deciding what information enters a model's context window and in what order. As windows grew, choosing what to include became more consequential than wording the instruction.

What RAG Actually Does

RAG (Retrieval-Augmented Generation) retrieves the most relevant passages from your documents and injects them into the prompt before the model answers.

Chunking: Why Split Size Matters

Documents are split into chunks before embedding.