What is Chunking?Chunking kya hai?

🧸 Think of it likeAise samjho

Like cutting a big pizza into slices so you can pick up exactly the piece you want.Badi pizza ko slices mein kaatne jaisa, taaki aap wahi tukda utha sako jo chahiye.

📘 ExplainedVistaar se

Models and vector search work best on focused passages. Chunks should hold a complete idea: split along headings, paragraphs and tables rather than every N characters, often a few hundred tokens with a small overlap, and store title, section, date and permissions as metadata. Test chunk sizes against real questions; bad chunking is a top cause of wrong RAG answers.Models aur vector search focused passages pe sabse accha kaam karte hain. Chunk mein poora idea hona chahiye: har N characters ke bajaye headings, paragraphs aur tables ke hisaab se todo, aksar kuch sau tokens thode overlap ke saath, aur title, section, date aur permissions metadata mein rakho. Asli sawaalon pe chunk sizes test karo; galat chunking galat RAG jawaabon ki badi wajah hai.

💡 Real exampleAsli example

A 40 page policy PDF becomes 120 chunks, one per section or paragraph group.40 page ki policy PDF 120 chunks ban jaati hai, har section ya paragraph group ka ek.

📖 Learn it step by stepStep by step seekho

• RAG: letting an LLM answer from your documentsRAG: LLM ko aapke documents se jawaab dene dena
• Embeddings and vector databases: searching by meaningEmbeddings aur vector databases: matlab se search karna

🔗 Related termsJude hue shabd

• RAGLetting an LLM answer from your documents: search for relevant passages first, then generate an answer from them with citations.LLM ko aapke documents se jawaab dene dena: pehle relevant hisse dhoondho, phir unhi se citations ke saath jawaab banao.
• EmbeddingA list of numbers that captures the meaning of text, images or items, so similar meanings are close together.Numbers ki list jo text, images ya items ka matlab pakadti hai, taaki milte julte matlab paas paas hon.
• Vector databaseA database built to store embeddings and quickly find the ones closest in meaning to a query.Database jo embeddings store karne aur query ke matlab ke sabse paas wale jaldi dhoondhne ke liye bana hai.