Diagnose a failure and justify whether to improve the prompt, evidence pipeline, training data, or ordinary application controls.
Diagnose before choosing a technique
A weak result can have several causes. First, check whether the system lacked instructions, relevant evidence, or the behavior needed for the task.
The observed failure should determine your next experiment. A missing policy document and inconsistent output structure need different investigations.
An evaluation baseline makes that experiment checkable. Keep examples of the original failure and define what a successful repair would look like.
Use prompting to express the task
Prompting provides instructions, examples, and context during an interaction. It can clarify the audience, scope, required fields, and evidence boundary.
A prompt repair fits an unclear request or missing instruction. For example, explicitly require an “unknown” field when the supplied notes omit a date.
A prompt limit appears when the necessary evidence is absent. Asking for greater accuracy does not supply a policy the application never retrieved.
Use retrieval to supply evidence
Retrieval-augmented generation selects external material and supplies it during the request. This can help an answer use current or task-specific information.
A retrieval repair fits a system that cannot find the appropriate source. Check collection coverage, permissions, versions, and passage selection before changing the model.
Evidence use still needs evaluation after retrieval succeeds. A correct passage in context does not guarantee a faithful final answer.
Understand what fine-tuning changes
Fine-tuning updates model parameters through additional training. It can adapt repeated behavior using examples or other training signals.
Training quality depends on the examples and objective you provide. Keep separate evaluation data to check whether improvements carry beyond the training material.
Technique choice is not a fixed ladder from prompting to retrieval to training. These methods address different gaps and can be combined when evidence supports the added complexity.
Choose a repair for a fictional assistant
A fictional event assistant must answer from current venue notes in a fixed three-field format. It sometimes uses last year’s room and omits an uncertainty field.
The room error first calls for checking source selection. The missing field calls for clearer instructions and an application check on the response structure.
A training experiment could become relevant if behavior remains inconsistent across representative cases. However, it would not replace current-source selection or permission checks.
| Observed gap | First investigation | Evidence that the repair helped |
|---|---|---|
| Current room absent from context | Source coverage and retrieval | Correct current passage reaches the model |
| Required field omitted | Instructions and output validation | Field appears with appropriate unknown values |
| Persistent task behavior fails despite clear context | Training examples and behavior evaluation | Held-out performance improves without critical regressions |
| Unauthorized publication | Application permissions and approval checks | Execution remains blocked without authority |
Practice: justify one change
Your exercise selects one row from the table and writes a repair proposal. Name the observed gap, proposed change, and test cases.
Your counterexample should reveal whether the repair only fits the original case. Include a missing source or a conflicting venue update.
A strong proposal connects the failure mechanism to measurable behavior. Tool availability varies; this exercise needs no training service or paid account.
Compare with the worked answer
Worked answer: For the outdated room, first ensure retrieval selects the approved current venue note. Test a current question, an explicitly historical question, and a question the notes cannot answer. Do not claim success from one corrected response or assume fine-tuning will keep venue facts current.
What to carry forward
Diagnose a failure and justify whether to improve the prompt, evidence pipeline, training data, or ordinary application controls.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.