Write a threat model and a safe test record that separates answer contamination, unauthorized actions, and evidence limits.
Recognize the authority problem
Prompt injection tries to redirect a model using instructions placed where the application expected task input or reference material.
Indirect injection can arrive through a retrieved document, message, or tool result. Although the assistant needs that material, its author cannot grant application permissions.
A trust boundary separates evidence from authority. Labeling source text can help communicate that distinction, but labels alone do not enforce it.
Trace one fictional attack
Our fictional assistant summarizes a product handbook for a support colleague. A retrieved appendix includes a sentence telling it to skip review and mark the case approved.
The attacker-controlled surface is the appendix text, not the colleague’s request or the application’s approval policy. The attempted change concerns authority, not product facts.
The intended outcome remains a source-grounded summary with no case update. Therefore, a convincing approval instruction inside the appendix must not authorize a write.
Map the boundaries
A small threat model names the asset, entry point, proposed action, and control outside the model. Start with one workflow rather than a universal security score.
Our protected assets are the case status and the reliability of the draft. Even without write access, a manipulated summary could still mislead its reviewer.
The boundary map separates answer quality from action authorization. Passing one check cannot establish the other.
| Stage | Risk to inspect | Control or evidence |
|---|---|---|
| Retrieve appendix | Untrusted instruction enters context | Record source and inspect selected content |
| Generate draft | Approval claim contaminates summary | Check claims against approved policy |
| Propose case update | Source text requests a write | Execution layer enforces user permissions |
| Review action | Approval hides changed destination or fields | Show exact action and arguments |
| Inspect outcome | Final refusal conceals an earlier write | Inspect tool log and dummy case state |
Put controls around the action
A minimal tool set limits the operations available to the assistant. A summary-only workflow may not need a case-update tool at all.
The execution layer must check caller permissions and proposed arguments before an action occurs. An instruction in retrieved text cannot expand that caller’s authority.
A review step should expose the concrete action, target, and fields. However, a person can also miss misleading context, so review is one layer rather than proof of safety.
Test safely with dummy state
A controlled rehearsal uses a fictional handbook and a dummy case store you own. Do not include real credentials or send tests to someone else’s system.
A comparison pair contains the same task with a clean appendix and with the injected approval instruction. Also include harmless quoted instructions to inspect unnecessary refusals.
An outcome record captures the draft, proposed calls, authorization decisions, and actual dummy-state changes. A refusal in the final message does not undo an earlier write.
Interpret the evidence carefully
A successful blocking result shows the observed attempt did not change dummy state under the recorded configuration. It does not prove resistance to every attack.
A useful regression set retains failed cases, ordinary tasks, versions, and repeat counts. Consequently, a repaired boundary can be checked without hiding lost usefulness.
A response plan isolates suspect source material, checks affected outputs, and restores the workflow only after relevant retests. Preserve necessary evidence with appropriate access controls.
Practice: investigate a misleading refusal
The fictional run ends with “I cannot approve this case.” However, its tool log shows an earlier update request, and dummy status changed to approved.
Your investigation identifies the failed boundary and distinguishes attempted action from completed action. Then propose a repair outside the final response wording.
Your retest checks the clean task, injected appendix, and harmless quotation. State separately whether the answer remained useful and whether unauthorized state changes occurred.
Compare with the worked answer
Worked answer: The action boundary failed: dummy state changed despite the later refusal. Remove the unnecessary write tool from a summary-only workflow, or enforce the caller’s authorization before execution where writes are genuinely required. Inspect the tool result and resulting state, not just the draft. Repeat the failed case and benign controls. A blocked write with a still-manipulated summary remains an answer-quality failure, so report those outcomes separately.
Use the method on your own task
Editable worksheet: Download the Markdown template and open it in a text editor. Use fictional or approved information; no account or paid service is required.
Download the worksheet (.md) ↓What to carry forward
Write a threat model and a safe test record that separates answer contamination, unauthorized actions, and evidence limits.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.