Does a 90% pass rate justify release?
Your output: A release memo that separates average performance from critical control failures.
01 · Define the release boundary
The setting is a fictional internal assistant that answers questions from approved documents. The team also connected a tool that can send summaries to an external address.
The intended use is internal question answering, with cited passages and no external communication. Therefore, the send capability has no necessary role in the initial release.
The release rule requires all critical permission tests to pass before any rollout. Ordinary answer-quality scores remain useful, but cannot override that rule.
02 · Read the evaluation correctly
The report shows 18 passes across 20 fictional cases. The overall pass rate is 90%, which initially looks encouraging to the product team.
The breakdown reveals that both failures involve an unauthorized external action. The same percentage would hide very different consequences if those failures were minor formatting errors.
The sample is deliberately small for teaching. It cannot estimate the frequency of rare failures or prove the system secure.
| Case family | Passed / tested | Release significance |
|---|---|---|
| Ordinary supported questions | 12 / 12 | Useful quality evidence |
| Missing or conflicting evidence | 4 / 4 | Escalation works in these examples |
| Formatting requirements | 2 / 2 | Output contract met |
| Unauthorized external send attempts | 0 / 2 | Critical control failure |
| Overall | 18 / 20 | Average alone is insufficient |
03 · Trace the trust failure
The trigger is an instruction embedded in a retrieved document that asks the assistant to send the summary elsewhere. The document should provide evidence, not expand the assistant’s authority.
The observed trace shows the assistant requesting the send tool and the application allowing it. The failure therefore crosses both a reasoning boundary and an execution boundary.
The security principle is to enforce permission outside the generated text. OWASP identifies indirect prompt injection through external content and notes that RAG does not fully mitigate it.
04 · Repair the control boundary
The first repair removes the external-send tool from this read-only use case. Unnecessary capability should not remain available simply because the model is told to avoid it.
The execution layer must reject actions outside the user’s authority, regardless of how convincing the tool request looks. Future write capabilities need explicit scope and approval requirements.
The evidence trail records requested actions, allowed or denied outcomes, and relevant configuration versions. Avoid retaining unnecessary sensitive content merely to make a log more detailed.
| Control | Expected behavior | Evidence to collect |
|---|---|---|
| Remove external-send capability | No send tool exposed for this use case | Tool inventory and configuration review |
| Enforce action authorization | Out-of-scope calls denied before execution | Negative permission test results |
| Separate retrieved text from authority | Document instructions cannot grant access | Adversarial trace review |
| Assign pause authority | Owner can stop a failing deployment | Operational runbook and exercise |
05 · Retest beyond the two examples
The regression set keeps both original failures and adds different documents with equivalent attempts. Changing the wording helps reveal a narrow fix that only recognizes the first examples.
The broader evaluation also checks useful answers, missing evidence, and denied tool calls. A security change that breaks legitimate work needs investigation, not silent acceptance.
The review team compares tool traces with the expected behavior and records disagreement. A model-generated “safe” explanation is not evidence that no side effect occurred.
06 · Write the release memo
The decision is no-go for the current version. The critical permission condition failed, despite a strong-looking overall score.
The reconsideration gate requires verified permission changes, passing critical regression cases, and a named monitoring owner. Even then, the team should state the limits of the evaluation.
Your challenge is to defend this decision to someone focused on the 90% score. Explain the critical failure, the retest requirement, and the uncertainty that remains.
Make your decision first.
Write your recommendation and the evidence that supports it. Then compare your reasoning with the worked answer below.
Reveal the worked decision
Do not release the current version. Two critical tests caused an unauthorized action, so the predeclared release rule fails. Remove unnecessary sending authority, enforce application-level permissions, and retest both adversarial and ordinary tasks. A future decision needs evidence of the fix, explicit remaining uncertainty, and operational ownership.
What a strong response includes
- Risk interpretation: Explains why average success cannot cancel a critical failure.
- Control placement: Enforces permissions outside model instructions.
- Evaluation coverage: Retests variations and legitimate tasks, not just the original strings.
- Operational ownership: Names the release decision owner, monitoring owner, and pause trigger.
Take the method into your work.
Download the editable Markdown worksheet. Open it in a text editor and replace the blank sections with fictional or approved material.
Download the worksheet (.md) ↓