Write a monitoring decision record with comparable samples, clear denominators, and a tested response.
Watch three different questions
Service health asks whether requests complete and how long they take. A quick, successful response can still contain a wrong answer.
Task quality asks whether the output meets the intended criteria. For a policy assistant, inspect current-source support and required exceptions.
Changing inputs can make yesterday’s evaluation less representative. Monitor relevant groups separately rather than assuming one average describes every request.
Define the denominator first
An error count needs a population and a time window. Ten wrong answers among fifty reviewed cases differs from ten among five thousand.
A reviewed sample is not automatically representative of all traffic. If reviewers mainly inspect complaints, its failure rate describes that selected group.
A useful dashboard note states the number reviewed, selection method, rubric version, and unresolved cases. Keep missing reviews visible instead of treating them as passes.
Investigate an invented quality drop
Our fictional policy assistant has two reviewed samples of one hundred answers each. The reviewers use the same rubric, but the mix of questions changes.
Routine questions pass at ninety-five percent in both samples. Exception questions pass at seventy percent in both, while their share grows.
The aggregate result falls from ninety to eighty percent without a within-group decline. The workload has changed; this evidence alone does not prove a model regression.
| Question group | Window A passed / reviewed | Window B passed / reviewed | Within-group pass rate |
|---|---|---|---|
| Routine | 76 / 80 | 38 / 40 | 95% in both |
| Exceptions | 14 / 20 | 42 / 60 | 70% in both |
| All reviewed answers | 90 / 100 | 80 / 100 | 90% then 80% |
Distinguish change from diagnosis
Input drift describes a change in the distribution of incoming data. In our example, exception requests become more common.
A quality regression means assessed behavior worsens relative to a defined comparison. Check comparable cases and configurations before attributing the aggregate change to the model.
A system change may involve the prompt, source index, permissions, or an upstream parser. Preserve version identifiers so the investigation can examine those alternatives.
Connect a signal to a response
An alert rule should state its owner, evidence, and next action. Choose thresholds for the use case instead of copying numbers from a teaching example.
Our fictional service owner might temporarily require review for all exception answers. That response addresses the weak group even though its measured rate has not changed.
A recovery check repeats the failing task types after a repair and reviews new live samples. Returning to a green service-health chart does not establish restored answer quality.
Avoid creating new problems in the logs
A useful event record can contain request ID, configuration version, relevant source IDs, check verdict, and timing. Collect only what the investigation actually needs.
Sensitive source text should not be copied into unrestricted diagnostic logs. Use approved access controls and retention rules for any necessary examples.
Delayed outcomes need a visible pending state. An unresolved customer case cannot yet establish success, even if the initial answer looked plausible.
Practice: write the monitoring decision
Your exercise recomputes both overall pass rates from the table. Explain why the total changes while the two group rates stay constant.
Your investigation plan requests comparable cases, source versions, and review-selection details. Name one observation that would contradict the workload-mix explanation.
Your response record separates observed evidence from suspected causes. Give an owner a specific action and a condition for checking recovery.
Compare with the worked answer
Worked answer: Window A is (76 + 14) / 100 = 90%; window B is (38 + 42) / 100 = 80%. Exception questions grow from 20% to 60% of the reviewed sample. Investigate the weak exception behavior and whether this sample reflects actual traffic. A drop on matched routine cases would contradict the claim that mix alone explains all deterioration. Do not retrain or roll back solely because the aggregate number changed.
Use the method on your own task
Editable worksheet: Download the Markdown template and open it in a text editor. Use fictional or approved information; no account or paid service is required.
Download the worksheet (.md) ↓What to carry forward
Write a monitoring decision record with comparable samples, clear denominators, and a tested response.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.