From Tool Use to RAG

Agents can now use tools to interact with the world. But what about the knowledge they need to do their job well? Most organizations have vast stores of documents, policies, and data that LLMs weren't trained on.

Retrieval-Augmented Generation (RAG) solves this — giving agents access to your organization's specific knowledge at query time.

What Is RAG? The Memory That Grounds AI

The Hallucination Problem

Large Language Models (LLMs) are trained on vast datasets, but they have a critical limitation: they can only draw on what they learned during training. Ask an LLM about your company's internal policies, last quarter's sales figures, or a document uploaded yesterday, and it will either admit ignorance or — worse — hallucinate a confident-sounding but fabricated answer.

This is the fundamental gap between a general-purpose AI and one that is genuinely useful inside your organization.

Before RAG: The Ungrounded LLM

Without retrieval, an LLM operates like a brilliant consultant who has never read any of your documents. It can reason, summarize, and generate fluent text, but every answer is drawn entirely from its pre-training knowledge. When that knowledge is insufficient, the model fills in the blanks with plausible-sounding fiction.

After RAG: The Grounded LLM

Retrieval-Augmented Generation (RAG) solves this by giving the model a memory — a searchable knowledge base of your actual documents. Before generating a response, the system retrieves the most relevant passages and injects them into the prompt as context. Now the LLM can cite real sources, reference actual data, and ground every claim in verifiable evidence.

Why RAG Matters for Agentic AI

In an agentic system, RAG becomes a tool the agent can invoke whenever it needs factual information. Rather than relying on parametric memory alone, the agent actively decides when to search, what to search for, and how to incorporate the results. This transforms RAG from a static pipeline into a dynamic, agent-driven capability — the difference between a fixed search box and an autonomous researcher who knows when and how to look things up.

The Core Promise

RAG lets you keep your LLM general-purpose while making it specific to your domain — without expensive fine-tuning, without retraining, and with full control over what knowledge the model can access.

The RAG Pipeline: From Documents to Answers

The Five Stages

Every RAG system follows the same fundamental pipeline, whether you build it from scratch or use a framework like LangChain. Understanding each stage is essential for building systems that return accurate, relevant answers.

Stage 1: Chunk

Raw documents — PDFs, web pages, Markdown files, database records — must be split into manageable pieces. Chunking strategies matter more than most teams expect. Too large, and retrieval returns irrelevant noise alongside the useful passage. Too small, and you lose the surrounding context that makes the passage meaningful.

Common strategies include fixed-size windows (e.g., 512 tokens with 50-token overlap), semantic chunking (splitting at paragraph or section boundaries), and recursive splitting that tries progressively smaller separators until chunks fit the target size.

Stage 2: Embed

Each chunk is converted into a vector embedding — a dense numerical representation that captures semantic meaning. Two passages about the same concept will have similar embeddings even if they use different words. This is what makes semantic search possible: you search by meaning, not by keywords.

Stage 3: Index

Embeddings are stored in a vector database optimized for similarity search. The index structure (HNSW, IVF, or flat brute-force) determines the trade-off between search speed and accuracy. For most applications, approximate nearest neighbor (ANN) algorithms provide sub-millisecond retrieval across millions of vectors.

Stage 4: Retrieve

When a user asks a question, their query is embedded using the same model, and the vector database returns the top-K most similar chunks. This is the critical filtering step — retrieval quality directly determines answer quality. Poor retrieval means the LLM never sees the right information, no matter how capable it is.

Stage 5: Generate

The retrieved chunks are injected into the LLM prompt as context, typically in a structured format: "Based on the following documents, answer the user's question." The model synthesizes the retrieved information into a coherent response, ideally citing which source each claim comes from.

The Pipeline as a Whole

Each stage has its own failure modes and optimization levers. The best RAG systems treat this as an end-to-end optimization problem, measuring retrieval precision, answer faithfulness, and user satisfaction as a unified system rather than tuning each stage in isolation.

Safety Considerations for RAG

RAG systems introduce their own category of risks. You're connecting an AI to your organization's documents — and that comes with responsibility.

PII in Retrieved Documents

Retrieved documents may contain personally identifiable information — names, email addresses, social security numbers, salary data. If this content enters the LLM's context, it can appear in responses to users who shouldn't see it. Implement PII detection and redaction in your retrieval pipeline, and enforce document-level access controls.

Data Quality: Garbage In, Garbage Out

A RAG system is only as good as its corpus. Outdated policies, draft documents mistakenly indexed, or contradictory information across sources will produce unreliable answers. Content curation and lifecycle management are not optional — they're prerequisites for a trustworthy system.

Hallucination Despite Retrieval

RAG reduces hallucination but doesn't eliminate it. An LLM can still generate claims that aren't supported by the retrieved documents, or subtly misinterpret what it retrieves. Always require source citations so users can verify, and consider adding faithfulness checks that validate generated answers against retrieved context.

Access Control: Who Sees What

Not everyone in your organization should see every document. Your RAG system must respect existing access controls — an intern querying the knowledge base shouldn't receive answers grounded in board-level financial documents. Implement user-aware retrieval that filters documents based on the requester's permissions.

Content Lifecycle Management

Documents age. Policies get updated. Last year's pricing guide is wrong today. Without a process for reviewing, updating, and archiving content in your RAG corpus, answer quality will silently degrade over time. Treat your knowledge base as a living system, not a one-time upload.

Key Takeaways

Before you move on, here's what to remember from this section:

• RAG grounds LLM responses in real documents — solving hallucination by retrieving facts at query time
• The pipeline flows: chunk → embed → index → retrieve → generate — each stage has tunable trade-offs
• Hybrid search combines semantic vectors with keyword matching (BM25) for the best of both worlds
• Reranking re-scores retrieved results with a more accurate model, improving final relevance
• Vector databases (Pinecone, Weaviate, Chroma) store and search embeddings at scale — choose based on your operational needs
• Content lifecycle matters — stale documents degrade retrieval quality over time