Imagine asking a brilliant new hire about your company's notice period on their first morning. They are articulate and well read, but they have not seen your HR handbook. If they feel they must answer anyway, they will produce something that sounds like a notice-period policy, perhaps "30 days, as is common", stated with complete confidence. That is roughly the position a large language model is in whenever you ask it about something it was never trained on.
Three limits of a closed-book model
A language model learns everything it knows during training, from a large but fixed body of mostly public text. Three practical limits follow.
- It cannot see private data. Internal policies, Confluence pages, contracts, support tickets and design documents were never in the training data, so the model has no way of knowing what they say.
- Its knowledge is frozen. Training stops at a cut-off date. A policy that changed last month, a product released this week or a price updated this morning does not exist for the model.
- It guesses when unsure. At heart, a language model predicts a plausible next token (From Logits to Tokens shows exactly how). Plausible is not the same as true. When the model lacks a fact it still produces fluent text, and a fluent wrong answer is a hallucination. In a casual chat that is a nuisance. In a bank, a hospital or a legal team, a single confident wrong answer has a real cost.
The fix is not to make the model remember more. It is to change the exam from closed-book to open-book: find the relevant pages first, put them in front of the model, and ask it to answer from those pages.
Retrieval, augmentation, generation
The name describes the three steps exactly.
- Retrieval: given the user's question, search your knowledge (PDFs, web pages, wikis, slides, databases) and fetch the few passages most likely to contain the answer.
- Augmentation: build a prompt that combines the retrieved passages with the question, together with an instruction such as "answer only from the context below".
- Generation: the language model reads that augmented prompt and writes the answer.
The model's job changes in an important way. It no longer has to know the answer. It has to read the answer from text it has been given, and summarising, extracting and rephrasing are tasks language models do very well. Retrieval also tends to bring back more text than the question needs, so the generation step serves as a filter too, keeping what is relevant and ignoring the rest.
The two halves of a RAG system
A RAG system has two phases that run at very different times.
Indexing (offline, ahead of time). You prepare the knowledge once, and again whenever it changes:
- Load the documents: PDFs, Word files, HTML, Markdown, slides.
- Split them into small passages called chunks. A whole document is too large to send to the model for every question, and too broad for search to match precisely.
- Embed each chunk: turn it into a vector, a list of numbers that captures its meaning, so that passages with similar meaning end up close together.
- Store the vectors, together with the original text and useful metadata, in a vector database.
Querying (online, per question). When a user asks something:
- Embed the question with the same embedding model.
- Search the vector database for the chunks whose vectors are closest to the question's vector, usually the top 3 to 10.
- Augment the prompt with those chunks.
- Generate the answer.
Meaning, not keywords
Suppose the HR policy contains a line about "10 sick-leave days annually". A user types "I am unwell, what leave can I take?" The words "sick" and "leave days" never appear in the question, yet a basic RAG system still finds the right passage. This is because the search compares meanings rather than words: the embedding of "I am unwell" sits close to the embedding of "sick leave". This is the first quiet surprise of RAG, and Module 2 explains how it works. Module 4 then covers the cases where meaning-based search is not enough, such as product codes and error IDs, and how to cover them as well.
Where a basic RAG pipeline breaks
The pipeline above works well in a demo. In production it fails in predictable places, and each failure is the subject of a later module:
| Where it breaks | Symptom | Where we fix it |
|---|---|---|
| Chunking | The answer is split across two chunks, or buried in a huge one | Module 2 |
| Search | The right chunk exists but is ranked 40th | Modules 3 and 4 |
| Query | The question is vague, compound or uses different vocabulary | Module 4 |
| Measurement | Nobody can tell if a change made things better | Module 5 |
| Freshness | Old and new versions of a policy are both retrieved | Module 6 |
| Wrong source | The answer lives in a SQL table, not a document | Module 7 |
| Relationships | The answer needs facts joined across documents | Module 8 |
Keep this table in mind. The rest of the course goes through it row by row.