Write a pilot decision memo with a defensible calculation, unresolved evidence, and conditions for the next step.
Decide what the pilot must prove
A pilot decision asks whether a specific workflow deserves its next commitment. First, define the users, tasks, allowed actions, and conditions under review.
Our fictional team tests an assistant that drafts support replies. Every reply requires review; automatic sending remains outside the pilot.
The decision owner writes the acceptance conditions before inspecting results. Otherwise, an attractive result can quietly replace the original purpose.
Keep three questions separate
Task quality asks whether the workflow produces acceptable outcomes, including difficult cases. A fluent draft is not yet a completed support task.
Operating value asks whether the complete process improves enough to justify its resources. Therefore, include review, correction, maintenance, and fallback work.
Operational readiness asks who maintains sources, handles failures, and restores the previous process. A strong demonstration does not answer these ownership questions.
Calculate the whole process
The fictional comparison uses two matched batches of one hundred tasks, completed to the same acceptance standard. All times below are invented teaching values.
The pilot total includes corrected drafts and a final review. Consequently, its correction time must not be counted again elsewhere.
The evidence limit is important: these matched batches are not a randomized experiment. Differences in staff, task difficulty, and timing could still influence the result.
| Work component | Existing process | AI-assisted pilot |
|---|---|---|
| Preparation / initial drafting | 800 minutes | 300 minutes |
| Review and correction | Included in 800 minutes | 400 minutes |
| Source and workflow upkeep allocated to this batch | Included in 800 minutes | 150 minutes |
| Total comparable handling effort | 800 minutes | 850 minutes |
The calculation: 300 + 400 + 150 = 850 minutes. The pilot uses 50 extra minutes, or 6.25% more effort than 800 minutes.
The boundary: this is a labor-time comparison, not a full cost or return-on-investment estimate. Fees and initial setup remain separate.
Find the break-even assumption
The simplified model holds drafting at 300 minutes and upkeep at 150 minutes per hundred tasks. Thus, review must fall below 350 minutes to save time.
A proposed improvement reduces review to 250 minutes, producing a projected total of 700 minutes. That would save 100 minutes, or 12.5%, against the stated baseline.
The missing evidence is whether review can actually fall without reducing quality. Label this a scenario, not an observed saving.
Do not average away a failed gate
The fictional quality review finds eight initial drafts mishandling a required policy exception. Reviewers catch and correct all eight before sending.
The gate agreed beforehand requires those exceptions to survive in every case of this selected release set. Therefore, the initial drafts fail that local gate.
Human correction shows a review control worked in these runs. However, it does not establish that fewer reviewers or a larger workload would remain acceptable.
Choose among three next steps
Stopping can mean returning this workflow to its existing process while keeping useful learning. It need not mean abandoning every possible AI use.
Improving requires a specific change, bounded effort, a retest, and a deadline. Otherwise, repeated pilots can postpone the decision indefinitely.
Scaling extends a defined boundary after evidence supports it. Increase one dimension at a time, such as volume, rather than changing users, permissions, and task types together.
| Option | When it fits | Evidence to record |
|---|---|---|
| Stop this approach | Required gate cannot be met within the agreed effort | Reason, fallback owner, retained lessons |
| Improve within current boundary | A plausible repair has a bounded cost and test | Change, owner, retest cases, deadline |
| Scale a defined use | Quality, value and operating conditions are supported | Scope limit, monitoring, rollback rehearsal |
Set the next experiment and stop rule
Our proposed next step keeps mandatory review and repairs how policy exceptions reach drafts. Use fresh cases alongside the original failures to check the change.
The experiment budget might allow one additional review cycle for this fictional team. Its owner records actual handling time and quality outcomes separately.
A stop rule could end this approach if the agreed exception gate still fails after that cycle. The team must choose its own threshold and fallback before testing.
Prepare for a bounded expansion
A rollout record names the permitted audience, tasks, configuration, and owner. It also states which results are observed and which remain assumptions.
A rollback rehearsal shows how the team resumes the previous process and handles unfinished work. Disabling the assistant alone may leave a queue without an owner.
A monitoring contract defines the reviewed sample, task groups, pending cases, and response triggers. Recheck assumptions when workload or source policy changes.
Practice: challenge the launch proposal
Your scenario adds a manager’s request to triple volume and halve review time. The only new evidence is a faster initial draft.
Your memo chooses stop, improve, or scale for the defined workflow. Show the 850-minute total and the review-time break-even assumption.
Your challenge names one observation that would change your recommendation. Then identify who owns the next test and what happens if it fails.
Compare with the worked answer
Worked answer: Do not approve the proposed expansion on this evidence. The pilot currently takes 850 rather than 800 minutes and fails its agreed exception gate. A bounded improvement cycle is reasonable only with an owner, budget, and retest. Review below 350 minutes would improve the simplified time comparison, but only if equivalent quality remains supported. The projected 700-minute scenario is not evidence of achieved savings. Keep the current review boundary, test fresh exception cases, and rehearse the fallback before considering expansion.
Use the method on your own task
Editable worksheet: Download the Markdown template and open it in a text editor. Use fictional or approved information; no account or paid service is required.
Download the worksheet (.md) ↓What to carry forward
Write a pilot decision memo with a defensible calculation, unresolved evidence, and conditions for the next step.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.