Should this team expand the pilot?
Your output: A workflow playbook with a baseline, approval rules, and recovery behavior.
01 · Define the real task
The setting is a fictional equipment supplier handling product questions. Staff want help drafting answers from an approved product handbook.
The pilot creates reply drafts but leaves staff responsible for checking and sending them. The team initially measures drafting speed alone.
The success criterion should describe the complete job: a correct, approved answer reaching the right customer. Therefore, the team needs total handling time and quality measures.
02 · Rebuild the baseline
The sample contains 20 fictional tickets processed with each approach. The values below are invented averages per ticket, not measured customer results.
The apparent win is a three-minute reduction in drafting time. However, checking and correction increase enough to erase that saving.
The comparison includes the same stages in both approaches. It remains a small illustrative pilot, so it cannot establish reliable savings across future workloads.
| Stage | Manual approach | Initial AI pilot |
|---|---|---|
| Read and classify | 2 | 2 |
| Draft | 4 | 1 |
| Check and correct | 2 | 5 |
| Approve and send | 1 | 1 |
| Total | 9 | 9 |
03 · Find the actual failure
The incident involves a customer asking about a product version missing from the current handbook. The draft confidently borrows a policy from an older model.
The reviewer has no source reference beside the disputed sentence. As a result, they spend more time reconstructing the evidence than writing a fresh answer.
The diagnosis is a weak evidence handoff and missing escalation rule. Faster generation cannot fix a workflow that hides why an answer should be trusted.
04 · Redesign the handoffs
The revised input contains the product identifier and question, with unnecessary personal details removed. The retrieval step selects only approved documents for that version.
The draft contract requires a supporting passage for each policy statement. If the version or evidence is missing, the workflow creates an escalation note instead of a confident answer.
The sending boundary remains outside the drafting step. A staff member sees the destination, proposed reply, and evidence before approving the send.
| Step | Required result | Failure behavior |
|---|---|---|
| Classify | Product/version identified | Ask staff to resolve missing version |
| Retrieve | Approved matching source | Escalate when evidence is absent |
| Draft | Reply plus passage references | Reject unsupported policy statements |
| Review | Named approval or rejection | Return for correction |
| Send | Approved destination and ticket ID | Check send status before retrying |
05 · Test recovery behavior
The duplicate case arrives when a send request times out after the remote service accepted it. Retrying blindly could send the same answer twice.
The recovery rule checks the ticket’s recorded send status or a supported idempotency mechanism. Uncertain status goes to a person instead of an unbounded retry loop.
The test set also includes missing product versions, contradictory documents, and questions outside the handbook. The correct outcome for some cases is escalation, not an answer.
06 · Decide what happens next
The decision is a limited second pilot, not a full rollout. The first design showed no total time saving and created avoidable checking work.
The next measurement tracks total handling time, unsupported statements, escalation rate, and reviewer effort. Compare matched task types and report the sample size with the results.
Your challenge is to write the stop rule and name its owner before the next pilot. Explain the failure response and the evidence required to expand.
Make your decision first.
Write your recommendation and the evidence that supports it. Then compare your reasoning with the worked answer below.
Reveal the worked decision
Do not expand the initial design. Its illustrative total is nine minutes per ticket in both approaches, and the missing-source case exposes a quality problem. Run a bounded second pilot only after adding visible evidence, escalation, approval, and duplicate-send handling. Treat expected savings as a hypothesis until measured.
What a strong response includes
- Complete baseline: Measures the same end-to-end stages in both approaches.
- Evidence handoff: Reviewers can inspect the source for a policy claim.
- Recovery: Missing evidence and uncertain send status have explicit owners.
- Decision: Rollout depends on observed quality and total effort, not drafting speed.
Take the method into your work.
Download the editable Markdown worksheet. Open it in a text editor and replace the blank sections with fictional or approved material.
Download the worksheet (.md) ↓Sources & further reading
Anthropic: Building effective agents — workflow architecture ↗