Lesson · Hinglish · ⏱ 8 min padhne mein
🧪 AI ko evaluate karna: kaise pata chale ki sach mein kaam karta hai
EnglishHinglish
🏠 Asli zindagi ki misaal
Restaurant nayi dish ko sirf chef ki raay se judge nahi karta. Woh bahut saari plates pe ek fixed checklist (taste, temperature, presentation) ke saath tasting sessions karte hain. Evals AI ke tasting sessions hain: asli sawaalon ka fixed set, saaf scoring ke saath, jo har baar prompt ya model badalne pe chalaaya jaata hai.
💡 Aasaan shabdon mein
LLM outputs badalte rehte hain, isliye 'demo mein accha laga' koi saboot nahi hai. Eval set banao: 30 se kuch sau tak asli inputs, expected answers ya grading rules ke saath. Jahan ho sake automatically score karo (exact match, sahi citation hai ya nahi, valid JSON), open-ended quality ke liye clear rubric ke saath LLM-as-judge use karo, aur haath se bhi spot-check karo. Har change pe score track karo, AI ke unit tests ki tarah. Agents ke liye task success rate, steps aur cost naapo. Evals hi production AI engineers ko demo banaane walon se alag karte hain.
🪜 Step by step
- Likho aapke feature ke liye accha ka matlab kya hai, acche aur bure outputs ke examples ke saath.
- 50 se 200 asli inputs ikattha karo: common cases, edge cases aur purane failures.
- Har ek ko score karne ka tarika chuno: jahan ho sake code checks (valid JSON, sahi category), open answers ke liye rubric.
- Rubric ke liye LLM judge tabhi use karo jab sample pe check kar lo ki woh human ratings se sehmat hai.
- Har prompt ya model change pe eval chalao, waqt ke saath score track karo, aur har naya production failure set mein jodo.
🔑 Zaroori shabd
- Eval set
- Test inputs ka fixed collection, expected results ke saath.
- LLM-as-judge
- Rubric ke saath ek LLM se doosre model ke jawaab grade karwana.
- Regression
- Yahan: aisa change jo pehle ke acche jawaabon ko kharab kar de.
👀 Example dekho
Eval set: 50 real HR questions
Check 1: answer cites a source → 47/50 ✅
Check 2: judge rubric 'correct & complete' → 42/50
Check 3: says 'not found' when it should → 5/6
Prompt v2 → v3: 42 → 46 correct. Ship v3 🚀Change ship hoga ya nahi, yeh feeling nahi, numbers decide karte hain.
⚠️ Aam galtiyaan jinse bachna hai
- Haath se kuch prompts try karke test karna ('vibe checks'). Jo aapne try karne ka socha nahi woh regressions chhoot jaate hain.
- Sirf aasaan examples wala eval set. Mushkil aur adversarial wale bhi daalo.
✅ Jaldi se check karo
Aapne prompt badla aur jo kuch jawaab try kiye woh behtar lag rahe hain. Aage kya?
- Turant ship karo
- Model se poochho ki behtar hua ya nahi
- Poora eval set chalao aur scores compare karo
Jawaab dekho
C. Poora eval set chalao aur scores compare karo: Kuch examples doosri jagah ke regressions chhupa sakte hain. Eval set asli asar dikhaata hai.
🙋 Seekhne wale yeh poochhte hain
Kitne examples chahiye?
50 acche examples se shuru karo; inse hi zyaadatar regressions pakad mein aa jaate hain. Waqt ke saath har important category ke liye sainkdon tak badhao.
Online evals kya hain?
Launch ke baad asli usage ke signals: thumbs up ya down, AI output mein edits, insaanon tak escalations, task completion.
📌 Yaad rakho
Evals nahi, to bharosa nahi. Har change naapo.
📖 Is lesson ke shabd (AI dictionary)
Poora course, aapke roadmap ke hisaab se
Progress tracking, quick checks aur step-by-step order ke saath. Yeh lesson in courses ka hissa hai:
AI Product Engineer · LLM / Agent Engineer · AI Forward Deployed Engineer · AI Product Manager
Course mein seekho