Produce a failure diagnosis supported by a source passage, retrieved results, and a targeted retest.
Start with the question and source
A wrong answer does not reveal which component failed. First, save the original question and the answer exactly as they appeared.
The expected evidence must exist in the approved collection. If no current document answers the question, changing search settings cannot create the missing fact.
A useful diagnosis compares the question, available source, retrieved passage, and generated claim. Keep these four items together while investigating.
Inspect the retrieved passages
Retrieval inspection asks whether the needed passage reached the answer stage. Record its title, version, access scope, and relevant wording.
Missing context can make an isolated passage misleading. For example, a deadline without its product name or exception may support the wrong answer.
Chunk design should preserve the information needed to interpret a passage. Therefore, test boundaries around headings, tables, definitions, and exceptions.
Separate freshness from relevance
Relevant wording can still come from an outdated source. A retired handbook may match the question better than a shorter current notice.
Version handling requires an explicit rule for current and historical questions. Keep retired documents distinguishable instead of treating every matching passage as equally authoritative.
Conflicting evidence should remain visible until resolved. Do not let a fluent answer silently merge incompatible policies into a new rule.
Work through a fictional policy failure
The fictional question asks when equipment must be returned after a loan ends. The current policy says seven days; the retired version says fourteen.
The retrieved evidence contains only the retired policy. The answer repeats fourteen days faithfully, so the first observed failure precedes generation.
The targeted repair is to correct current-version selection and rerun the question. Then test an explicitly historical question to preserve legitimate archive use.
| Observed evidence | Likely failure location | Next check |
|---|---|---|
| Current policy absent from collection | Source coverage | Add an approved current source |
| Current policy present but not retrieved | Retrieval or filtering | Inspect query, metadata, and returned passages |
| Correct passage retrieved without its exception | Context preparation | Check chunk boundaries and surrounding text |
| Complete correct passage supplied; answer contradicts it | Answer generation | Inspect instructions and claim support |
| Conflicting current sources supplied | Source governance | Resolve authority or state the conflict |
Retest both sides of the boundary
A focused retest checks the original failure after the repair. However, one corrected answer does not show that nearby cases still work.
Boundary questions cover different product names, dates, exceptions, and absent information. Include a question whose correct response is “the collection does not say.”
Separate results show whether evidence was found and whether the answer used it correctly. This prevents a good final guess from hiding failed retrieval.
Practice: write a diagnosis memo
Your exercise uses three invented policy snippets: current, retired, and an exception. Write four questions that need different evidence.
A manual simulation is enough to practice the reasoning. Select passages yourself, draft answers, and deliberately remove one crucial exception.
A strong diagnosis names the first observed break and supplies supporting evidence. Propose one repair and a question that could disprove its success.
Compare with the worked answer
Worked answer: If the exception was removed before drafting, diagnose context preparation first. Restore the complete rule, then test both an ordinary loan and an exception. If both passages are supplied but the answer still ignores the exception, investigate generation next.
What to carry forward
Produce a failure diagnosis supported by a source passage, retrieved results, and a targeted retest.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.