Lesson · ⏱ 7 min read
⚡ Serving LLMs efficiently: GPUs, batching and cost
EnglishHinglish
🏠 Real-life example
An auto-rickshaw carrying one passenger at a time is slow and expensive per person. A bus that waits a few seconds to fill up moves far more people for the same fuel. LLM servers 'batch' many user requests together on one GPU for the same reason: much better throughput and lower cost per request.
💡 In simple words
Self-hosting open models (Llama, Qwen, Mistral) needs GPUs, and GPUs are expensive. Inference servers like vLLM or TGI maximise GPU use with continuous batching (adding new requests mid-flight) and efficient memory for the conversation cache. Key metrics: time to first token (how fast the reply starts), tokens per second, p95 latency and cost per 1,000 requests. Quantisation shrinks models to fit cheaper GPUs, caching repeated prompts saves money, and an LLM gateway in front routes traffic, enforces budgets and falls back between providers.
🪜 Step by step
- Pick the smallest model that meets your quality bar on your eval set.
- Consider quantisation (8 bit or 4 bit) to fit on cheaper GPUs with little quality loss; measure it.
- Serve with an LLM inference engine (vLLM, TGI) for continuous batching and efficient KV cache memory.
- Measure time to first token, tokens per second and cost per 1,000 requests under realistic load.
- Autoscale on queue length, cache repeated prompts, and route easy requests to smaller models.
🔑 Key terms
- Batching
- Processing many requests together to use the GPU fully.
- Time to first token
- How quickly the first words of a reply appear.
- p95 latency
- 95% of requests are faster than this. It captures the slow tail.
👀 See it in action
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192
# → OpenAI-compatible API on :8000 with continuous batching built in
# Track: time-to-first-token, tokens/sec, p95 latency, GPU utilisation, ₹ per 1k requestsOne command gives you a batched, production-style LLM server.
⚠️ Common mistakes to avoid
- Comparing GPUs only by price per hour. Compare cost per request at your real traffic.
- Self hosting by default. At low volume, an API is often cheaper and far simpler.
✅ Quick check
Your GPU sits at 20% utilisation while users wait. What helps most?
- A bigger model
- An inference server with continuous batching
- Buy more GPUs
Show answer
B. An inference server with continuous batching: Batching packs more requests onto the same GPU, raising utilisation and throughput.
🙋 Questions learners ask
What is batching and why does it help?
Processing many requests together keeps the GPU busy. Continuous batching adds new requests into a running batch instead of waiting.
When should we self host an LLM?
When data can't leave your environment, when volume is high enough that it's cheaper, or when you need a fine tuned open model.
📌 Remember
Batch requests, measure latency and cost, and quantise and cache where you can.
📖 Words from this lesson (AI dictionary)
Learn this as part of a full course
Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:
Continue in the course