Lesson · ⏱ 9 min read
📚 RAG: letting an LLM answer from your documents
EnglishHinglish
🏠 Real-life example
An open-book exam: you don't memorise the whole textbook. When a question comes, you find the right pages, read them, and write your answer from them, citing the chapter. RAG (Retrieval-Augmented Generation) makes an LLM take an open-book exam with your company's documents.
💡 In simple words
There are two phases. Indexing: split documents into chunks (a few paragraphs each), create an embedding for each chunk, and store them in a vector database. Answering: embed the user's question, retrieve the most relevant chunks, put them into the prompt ('Answer only from these sources…'), and let the LLM write the answer with citations. RAG keeps answers current and grounded in your data without retraining the model. Most RAG quality problems are retrieval problems, such as bad chunking or missing documents, so measure retrieval separately from the final answer.
🎬 See how it works
Indexing happens once: documents → chunks → embeddings → vector DB. For each question, retrieve the best chunks, add them to the prompt, and answer with the source.
🪜 Step by step
- Collect and clean the documents: remove duplicates, old versions and junk text.
- Chunk them along headings and paragraphs, a few hundred tokens each, and keep title, section and date as metadata.
- Embed the chunks and store them; for each question, retrieve the top matches (ideally hybrid search plus a reranker).
- Prompt: "Answer only from the sources below and cite them. If they don't contain the answer, say so."
- Build an eval set of real questions; measure retrieval (was the right chunk found?) and answers (correct and faithful?) separately.
🔑 Key terms
- Chunking
- Splitting documents into small, searchable pieces.
- Retrieval
- Finding the most relevant chunks for a question.
- Grounding / citations
- Answering only from the retrieved sources and showing which ones.
👀 See it in action
question = "How many casual leaves do new joiners get?"
chunks = vector_db.search(embed(question), top_k=4) # retrieve
context = "\n\n".join(f"[{c.source}] {c.text}" for c in chunks)
prompt = f"Answer ONLY from the sources below. Cite them. If not found, say so.\n\n{context}\n\nQ: {question}"
answer = llm(prompt) # → "New joiners get 8 casual leaves per year [HR-Policy-2026 §4.2]"Retrieve → augment the prompt → generate. Plus an honest 'not found' rule.
⚠️ Common mistakes to avoid
- Fixed size chunks that cut tables and lists in half. Respect document structure.
- Stuffing 30 chunks into the prompt. More isn't better; a few relevant chunks usually beat many.
- Ignoring permissions, so users see documents they shouldn't. Filter at retrieval time.
✅ Quick check
Your RAG bot gives wrong answers. The right document exists, but it's never retrieved. What should you fix first?
- Increase temperature
- Switch to a bigger LLM
- Retrieval: chunking, embeddings or hybrid search
Show answer
C. Retrieval: chunking, embeddings or hybrid search: If the right text never reaches the prompt, no LLM can answer correctly. Fix retrieval first.
🙋 Questions learners ask
RAG or a bigger context window?
Even with huge context windows, RAG is usually cheaper, faster and more accurate, because the model sees only what's relevant. For one small document, just put it in the prompt.
How do I keep the index up to date?
Run a sync job that re-embeds changed documents and deletes removed ones, triggered by changes or on a schedule.
📌 Remember
RAG = search your docs, then answer from them with citations.
📖 Words from this lesson (AI dictionary)
Learn this as part of a full course
Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:
Data Engineer (AI focus) · AI Product Engineer · LLM / Agent Engineer · AI Forward Deployed Engineer
Continue in the course