Lesson · ⏱ 7 min read

📋 Writing an AI product spec (PRD) with an eval plan

Q: What's the key section an AI PRD needs that a normal PRD often lacks?

An eval plan and quality bar for AI outputs. AI outputs vary, so you must define in advance how quality is measured and what's good enough.

🏠 Real-life example

When you hire a contractor to renovate a kitchen, a good brief says what you want, the budget, the must-haves, what's off-limits and how you'll inspect the work. An AI PRD is that brief for an AI feature, with one extra part normal software doesn't need: how you'll judge answers that can be 'mostly right'.

💡 In simple words

A good AI PRD covers the user and problem, the success metric, scope and non-goals, and the user experience, including what happens when the AI is unsure or wrong. Add the AI-specific sections too: the data and context the model needs, the approach (prompting, RAG, fine-tuning), the eval plan (an eval set, quality bar, how it's measured), risks (hallucination, privacy, bias, prompt injection) and mitigations, cost and latency budgets, human oversight, and launch criteria. Writing the eval plan before building forces everyone to agree on what 'good' means.

🪜 Step by step

Problem and users: who has the pain, how big it is, and today's baseline.
Solution and scope: what the AI does, what it doesn't, and where humans stay in the loop.
Quality bar: the eval set, how it's scored, and the launch thresholds (including zero tolerance cases).
Risks and mitigations: wrong answers, privacy, abuse, cost; and the fallback for each.
Metrics and rollout: layered success metrics, guardrails, a phased launch plan and a review date.

🔑 Key terms

Non-goals: Things the feature deliberately won't do.
Quality bar: The minimum eval score needed to launch.
Launch criteria: Conditions that must be met before going live.

👀 See it in action

text

AI PRD: Policy assistant for branch staff
Problem:   staff spend ~25 min/day searching policy PDFs
Metric:    time-to-answer < 1 min; 90% answers correct (eval set of 100 real questions)
Approach:  model API + RAG over policy docs, citations required
Non-goals: no customer-facing use in v1; no advice outside the policies
Risks:     wrong answer → always cite + 'confirm with manager' for loans > ₹10L
Launch:    ≥ 90% on evals, < 3 s p95 latency, legal sign-off

The eval set and quality bar are decided before any code is written.

⚠️ Common mistakes to avoid

Writing "the AI should be accurate" with no measurable definition.
Leaving out what happens when the AI fails. That's often the most important part of the spec.

✅ Quick check

What's the key section an AI PRD needs that a normal PRD often lacks?

A list of competitors
An eval plan and quality bar for AI outputs
A colour palette

Show answer

B. An eval plan and quality bar for AI outputs: AI outputs vary, so you must define in advance how quality is measured and what's good enough.

🙋 Questions learners ask

How is an AI PRD different from a normal PRD?

It adds an eval plan and quality thresholds, explicit failure handling, data needs, cost per use and model change management.

Who writes the eval set?

Usually the PM with domain experts and engineers together. Domain experts define what a correct answer is.

📌 Remember

Problem, metric, scope, UX for failure, eval plan, risks, launch bar.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

AI Product Manager

Continue in the course