What is Latency?Latency kya hai?

🧸 Think of it likeAise samjho

Like the time between ordering a chai and getting it in your hand.Chai order karne aur haath mein aane ke beech ka time.

📘 ExplainedVistaar se

Measure percentiles, not just averages: p95 latency is the time within which 95% of requests finish. For LLMs, track time to first token (how soon text appears) and tokens per second. Reduce latency with streaming, shorter prompts, caching, smaller models and batching.Sirf average nahi, percentiles naapo: p95 latency woh time hai jisme 95% requests poori hoti hain. LLMs ke liye time to first token (text kitni jaldi dikhta hai) aur tokens per second track karo. Streaming, chhote prompts, caching, chhote models aur batching se latency kam karo.

💡 Real exampleAsli example

p95 latency of 800 ms means 95 of 100 requests finish within 0.8 seconds.p95 latency 800 ms matlab 100 mein se 95 requests 0.8 second mein poori hoti hain.

📖 Learn it step by stepStep by step seekho

• Serving LLMs efficiently: GPUs, batching and costLLMs ko efficiently serve karna: GPUs, batching aur cost
• Serving a model as an API with FastAPIModel ko FastAPI se API ke roop mein serve karna

🔗 Related termsJude hue shabd

• StreamingShowing an LLM's answer word by word as it's generated, instead of waiting for all of it.LLM ka jawaab poora hone ka intezaar karne ke bajaye shabd ba shabd dikhana jaise jaise banta hai.
• InferenceUsing a trained model to make predictions on new data.Trained model se naye data pe predictions karna.
• AutoscalingAutomatically adding or removing servers as traffic goes up or down.Traffic badhne ya ghatne pe apne aap servers jodna ya hataana.