What is KV cache?KV cache kya hai?

🧸 Think of it likeAise samjho

Like keeping notes while reading a long story, so for each new page you don't have to reread the whole book.Lambi kahaani padhte waqt notes banaane jaisa, taaki har naye page ke liye poori kitaab dobara na padhni pade.

📘 ExplainedVistaar se

Generating text one token at a time would otherwise redo work for all earlier tokens at each step. The KV cache makes generation much faster but uses GPU memory that grows with context length and batch size, which inference engines manage with techniques like paged attention.Ek ek token generate karne mein warna har step pe pichhle saare tokens ka kaam dobara hota. KV cache generation kaafi tez karta hai par GPU memory leta hai jo context length aur batch size ke saath badhti hai, jise inference engines paged attention jaise tarikon se sambhaalte hain.

💡 Real exampleAsli example

Long chats use more GPU memory per user because the KV cache grows.Lambi chats mein har user ki zyaada GPU memory lagti hai kyunki KV cache badhta hai.

📖 Learn it step by stepStep by step seekho

• Transformers and attention: how LLMs readTransformers aur attention: LLMs kaise padhte hain
• Serving LLMs efficiently: GPUs, batching and costLLMs ko efficiently serve karna: GPUs, batching aur cost

🔗 Related termsJude hue shabd

• AttentionThe mechanism in transformers that lets each word look at every other word to understand its meaning in context.Transformers ka woh mechanism jisse har shabd baaki har shabd ko dekhkar context mein apna matlab samajhta hai.
• InferenceUsing a trained model to make predictions on new data.Trained model se naye data pe predictions karna.
• GPUA chip that does thousands of maths operations in parallel, which makes training and running AI fast.Chip jo hazaaron maths operations ek saath karta hai, jisse AI ki training aur chalaana tez hota hai.