Lesson · Hinglish · ⏱ 7 min padhne mein

🆎 AI features ke liye metrics aur A/B testing

Q: A/B test mein users ko random tarah se kyun baantte hain?

Taaki groups ke beech sirf feature ka systematic fark ho. Randomisation baaki sab balance kar deta hai, isliye fark feature ki wajah se maana ja sakta hai.

🏠 Asli zindagi ki misaal

Chai stall wala sochta hai ki nayi adrak wali recipe zyaada bikegi ya nahi. Poori tarah badal kar pichhle mahine se compare karna galat hoga, kyunki mausam aur tyohaar sales badal dete hain. Iski jagah do hafte tak woh aadhe customers ko random tarah se purani recipe deta hai aur aadhon ko nayi, phir compare karta hai. Yahi A/B test hai.

💡 Aasaan shabdon mein

AI feature launch karne se pehle uska success metric define karo (resolution rate, time bachaya, conversion), guardrail metrics jo kharab nahi hone chahiye (complaints, cost, latency), aur kitna improvement matter karta hai. A/B test users ko random tarah se control (purana) aur treatment (naya) groups mein baantta hai, taaki dono mein systematic fark sirf feature ka ho, aur aap causation claim kar sako. Itni der chalao ki statistical significance mile, aur beech mein jhaank kar jaldi band mat karo. AI ke liye launch se pehle offline evals aur launch ke baad online A/B results, dono milaao.

🎬 Dekho kaise kaam karta hai

Users random tarah se baante jaate hain: aadhe purana version (A) dekhte hain, aadhe naya (B). Split random hai, isliye results ka fark feature ki wajah se hai.

🪜 Step by step

Hypothesis likho: "AI reply drafts handling time 15% kam karenge, satisfaction giraaye bina."
Ek primary metric aur kuch guardrail metrics chuno jo bigadne nahi chahiye.
Users (ya agents) ko randomly control aur treatment mein baanto, aur andaaza lagao kitne chahiye aur kitne din.
Poori planned duration chalao, beech mein jhaank kar jaldi band mat karo.
Confidence intervals ke saath fark analyse karo, guardrails aur segments check karo, phir decide karo.

🔑 Zaroori shabd

Control / treatment: Bina change wala group vs change wala group.
Guardrail metric: Metric jo kharab nahi hona chahiye jab aap doosra optimise karo.
Statistical significance: Bharosa ki fark sirf random kismat nahi hai.

👀 Example dekho

text

Feature: AI-suggested replies for support agents
Success:   handling time  9.1 → 6.8 min (treatment) ✅ significant
Guardrail: CSAT           4.4 → 4.4                 ✅ unchanged
Guardrail: cost/ticket    +₹1.2                     ✅ within budget
Decision: roll out to 100%

Ek success metric, guardrails, aur saaf go / no-go faisla.

⚠️ Aam galtiyaan jinse bachna hai

Results acche dikhte hi band kar dena. Jaldi jhaankne se jhoothi jeet milti hai.
Bahut saare metrics test karna aur jo hila uska jashn manana. Primary metric pehle tay karo.

✅ Jaldi se check karo

A/B test mein users ko random tarah se kyun baantte hain?

Taaki groups ke beech sirf feature ka systematic fark ho
Test chhota karne ke liye
Code karna aasaan hai

Jawaab dekho

A. Taaki groups ke beech sirf feature ka systematic fark ho: Randomisation baaki sab balance kar deta hai, isliye fark feature ki wajah se maana ja sakta hai.

🙋 Seekhne wale yeh poochhte hain

Statistical significance simple shabdon mein kya hai?

Agar change ka koi asli asar na hota to yeh fark kitna unlikely hota. Yeh nahi batata ki asar bada ya keemti hai.

Agar randomise na kar sakein to?

Region ya team ke hisaab se phased rollout karo, pehle aur baad dhyaan se compare karo, aur imaandaari se maano ki saboot kamzor hai.

📌 Yaad rakho

Pehle success aur guardrails define karo, phir randomise karo aur significance ka intezaar karo.

📖 Is lesson ke shabd (AI dictionary)

Poora course, aapke roadmap ke hisaab se

Progress tracking, quick checks aur step-by-step order ke saath. Yeh lesson in courses ka hissa hai:

AI Product Manager

Course mein seekho