Write a model comparison memo with a controlled test, task-level results, trade-offs, and a bounded recommendation.
Define the decision you need
A model comparison should answer a specific operating question. State which task, users, evidence, and output standard the selected model must support.
A public benchmark may measure a different task or environment. Use it as background, then test whether the candidate fits your own work.
A hard constraint can eliminate an otherwise capable option. Define required data handling, tool permissions, and response limits before ranking answers.
Hold the right things constant
A controlled comparison uses the same task inputs, source material, and grading rules. Record model versions and available configuration settings.
Two valid questions require different designs: compare models under one shared setup, or compare separately optimized systems. State which question your experiment answers.
A fair tuning budget limits how much adaptation each candidate receives. If one gets extensive prompt refinement, label that difference in the result.
Reduce avoidable judging bias
Blind review can hide model names while outputs are graded. Rotate presentation order so the first answer does not always belong to one candidate.
A concrete rubric separates factual support, completeness, instruction following, and forbidden behavior. Attractive wording should not silently become the definition of quality.
Reviewer disagreement deserves investigation before aggregation. Check whether the task, expected answer, or rubric contains an ambiguity.
Inspect a fictional repeated-trial comparison
This fictional exercise tests three cases three times per candidate. A trial passes only when every stated check for that case passes.
Candidate A passes seven of nine trials; candidate B passes eight. Both candidates complete every trial on two cases.
The category results show where more testing is needed. B has more total passes here, but neither consistently handles the missing-evidence case.
| Case | Candidate A | Candidate B |
|---|---|---|
| Ordinary request | 3 / 3 | 3 / 3 |
| Missing evidence | 1 / 3 | 2 / 3 |
| Ambiguous request | 3 / 3 | 3 / 3 |
Include operational effort
Review effort may change the preferred option. A shorter response can still require more correction than a slower, more complete one.
A measured trade-off keeps quality, latency, and cost visible separately. Avoid combining them into one score unless the weighting matches a stated decision.
A bounded recommendation describes where the evidence applies. Small fictional samples cannot establish general reliability or identify a universally best model.
Practice: write the comparison memo
Your exercise uses the table to recommend the next test. Include the pass totals, unresolved failure category, and limits of the sample.
An additional test should target missing evidence with new examples and repeated trials. Keep the acceptance rule unchanged across candidates.
A strong memo distinguishes sample performance from production suitability. Explain what new evidence could change your recommendation.
Compare with the worked answer
Worked answer: B passes eight of nine trials, while A passes seven. Both pass every trial on two of three cases. Missing-evidence handling remains inconsistent, so test that category further before selecting either for work requiring dependable abstention. Review effort and other operating constraints remain unmeasured.
What to carry forward
Write a model comparison memo with a controlled test, task-level results, trade-offs, and a bounded recommendation.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.