Start with the problem your customer needs solved. AI is one possible approach. This playbook helps you decide whether its flexibility is worth its cost, uncertainty, and operating complexity.
WHAT YOU’LL LEAVE WITH
An opportunity brief, evaluation plan, and clear acceptance criteria.
Before you begin
- A clearly stated user problem
- Existing alternatives and their limitations
- Examples of successful and unacceptable behavior
STEP 01
Describe the problem without AI
State the task, user, context, and desired outcome. If the problem statement only makes sense when it mentions a chatbot, work backward to the underlying need.
STEP 02
Compare simpler alternatives
Consider a clearer interface, better content, search, or deterministic rules. Identify where ambiguous language or context creates a real need for AI.
STEP 03
Define acceptable behavior
Agree on task success, unacceptable errors, response time, and fallback behavior. Include design, engineering, and relevant business stakeholders.
STEP 04
Create an evaluation set
Collect representative cases and difficult edge cases. Write expected behavior and review criteria. Keep some examples separate for checking later changes.
STEP 05
Build a narrow experiment
Prototype the simplest viable approach with sample data. Show uncertainty and give users a way to recover or get human help. Do not hide the limits behind confident copy.
STEP 06
Measure the full task
Track successful completions, wrong answers, abstentions, escalation, latency, and cost per successful task. A cheap response is not useful if users need five attempts.
STEP 07
Choose a controlled next step
Use evidence to continue, revise, or stop. Before a limited rollout, agree on monitoring, ownership, rollback conditions, and how new failures enter the evaluation set.
MAKE THIS PROMPT YOUR OWN
Evaluate [proposed feature] against the user problem [problem]. Compare a better interface, deterministic rules or search, and an AI approach. List assumptions and required evidence. Propose task-level acceptance criteria, unacceptable failures, fallback behavior, and a small evaluation plan. Do not invent demand or performance results.
Check your work
- Is AI helping with a real source of ambiguity?
- Are acceptance criteria observable?
- Can users recover when the system is wrong?
- Do economics include failures and repeat attempts?
Further reading: official guidance / research ↗
Editorial guide · Reviewed September 27, 2026. Tool interfaces may change.