Lesson · Hinglish · ⏱ 7 min padhne mein
⚡ LLMs ko efficiently serve karna: GPUs, batching aur cost
EnglishHinglish
🏠 Asli zindagi ki misaal
Ek baar mein ek passenger le jaane wala auto slow aur per person mehenga padta hai. Bus thode second rukkar bharti hai aur utne hi fuel mein kahin zyaada log le jaati hai. LLM servers isi wajah se ek GPU pe bahut saari user requests ko 'batch' karte hain: kahin behtar throughput, aur har request ki cost kam.
💡 Aasaan shabdon mein
Open models (Llama, Qwen, Mistral) khud host karne ke liye GPUs chahiye, aur GPUs mehenge hain. vLLM ya TGI jaise inference servers continuous batching (chalti requests ke beech nayi requests jodna) aur conversation cache ke liye efficient memory se GPU ka poora use karte hain. Key metrics: time to first token (jawaab kitni jaldi shuru hota hai), tokens per second, p95 latency aur har 1,000 requests ki cost. Quantisation models ko chhota karke saste GPUs mein fit karta hai, repeat prompts ki caching paise bachaati hai, aur saamne laga LLM gateway traffic route karta hai, budgets lagaata hai aur providers ke beech fallback karta hai.
🪜 Step by step
- Woh sabse chhota model chuno jo aapke eval set pe quality bar paar kare.
- Saste GPUs pe fit karne ke liye quantisation (8 bit ya 4 bit) socho, quality thodi hi girti hai; naap ke dekho.
- Continuous batching aur efficient KV cache memory ke liye LLM inference engine (vLLM, TGI) se serve karo.
- Realistic load pe time to first token, tokens per second aur 1,000 requests ka cost naapo.
- Queue length pe autoscale karo, repeat prompts cache karo, aur aasaan requests chhote models pe bhejo.
🔑 Zaroori shabd
- Batching
- GPU ka poora use karne ke liye bahut saari requests ek saath process karna.
- Time to first token
- Jawaab ke pehle words kitni jaldi dikhte hain.
- p95 latency
- 95% requests isse fast hain. Slow wali requests isse pakdi jaati hain.
👀 Example dekho
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192
# → OpenAI-compatible API on :8000 with continuous batching built in
# Track: time-to-first-token, tokens/sec, p95 latency, GPU utilisation, ₹ per 1k requestsEk command se batched, production-style LLM server mil jaata hai.
⚠️ Aam galtiyaan jinse bachna hai
- GPUs ko sirf per hour price se compare karna. Apne asli traffic pe per request cost compare karo.
- Default mein self hosting. Kam volume pe API aksar sasti aur bahut simple hoti hai.
✅ Jaldi se check karo
Users wait kar rahe hain aur GPU sirf 20% use ho raha hai. Sabse zyaada kya madad karega?
- Bada model
- Continuous batching wala inference server
- Aur GPUs khareedo
Jawaab dekho
B. Continuous batching wala inference server: Batching same GPU pe zyaada requests daalta hai, jisse utilisation aur throughput badhte hain.
🙋 Seekhne wale yeh poochhte hain
Batching kya hai aur kyun madad karti hai?
Bahut saari requests ek saath process karne se GPU busy rehta hai. Continuous batching intezaar karne ke bajaye naye requests chalte batch mein jod deti hai.
LLM kab self host karein?
Jab data aapke environment se bahar na ja sake, jab volume itna ho ki sasta pade, ya jab fine tuned open model chahiye.
📌 Yaad rakho
Requests batch karo, latency aur cost naapo, aur jahan ho sake quantise aur cache karo.
📖 Is lesson ke shabd (AI dictionary)
Poora course, aapke roadmap ke hisaab se
Progress tracking, quick checks aur step-by-step order ke saath. Yeh lesson in courses ka hissa hai:
Course mein seekho