Lesson · ⏱ 7 min read

🆎 Metrics and A/B testing for AI features

🏠 Real-life example

A chai stall owner wonders if a new ginger recipe sells better. Switching completely and comparing with last month is unfair, because weather and festivals change sales. Instead, for two weeks they randomly serve the old recipe to half the customers and the new one to the other half, then compare. That's an A/B test.

💡 In simple words

Before launching an AI feature, define its success metric (resolution rate, time saved, conversion), guardrail metrics that must not get worse (complaints, cost, latency), and how big an improvement matters. An A/B test randomly splits users into control (old) and treatment (new) groups so that the only systematic difference is the feature, which lets you claim causation. Run it long enough to reach statistical significance, and don't peek and stop early. For AI, combine offline evals before launch with online A/B results after it.

🎬 See how it works

Users are split at random: half see the old version (A), half the new one (B). Because the split is random, the difference in results is caused by the feature.

🪜 Step by step

Write a hypothesis: "AI reply drafts will cut handling time by 15% without lowering satisfaction."
Pick one primary metric and a few guardrail metrics that must not get worse.
Randomly split users (or agents) into control and treatment, and estimate how many you need and for how long.
Run the full planned duration without peeking and stopping early.
Analyse the difference with confidence intervals, check guardrails and segments, then decide.

🔑 Key terms

Control / treatment: The group without the change vs the group with it.
Guardrail metric: A metric that must not get worse while you optimise another.
Statistical significance: Confidence that a difference isn't just random luck.

👀 See it in action

text

Feature: AI-suggested replies for support agents
Success:   handling time  9.1 → 6.8 min (treatment) ✅ significant
Guardrail: CSAT           4.4 → 4.4                 ✅ unchanged
Guardrail: cost/ticket    +₹1.2                     ✅ within budget
Decision: roll out to 100%

One success metric, guardrails, and a clear go / no-go call.

⚠️ Common mistakes to avoid

Stopping as soon as results look good. Early peeks produce false wins.
Testing many metrics and celebrating whichever moved. Decide the primary metric beforehand.

✅ Quick check

Why randomly assign users in an A/B test?

So the feature is the only systematic difference between groups
To make the test shorter
It's easier to code

Show answer

A. So the feature is the only systematic difference between groups: Randomisation balances everything else, so the difference can be attributed to the feature.

🙋 Questions learners ask

What is statistical significance in simple words?

How unlikely the difference would be if the change had no real effect. It doesn't tell you the effect is big or valuable.

What if we can't randomise?

Use a phased rollout by region or team, compare before and after carefully, and be honest that the evidence is weaker.

📌 Remember

Define success and guardrails first, then randomise and wait for significance.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

AI Product Manager

Continue in the course