Create a source register, identify a coverage gap, and justify a bounded decision about using the data.
Define the job before collecting data
Data readiness depends on the intended task. A document collection suitable for historical research may be unsuitable for answering current policy questions.
Our fictional assistant drafts answers about returns using an approved product handbook. However, it cannot approve refunds or change customer records.
The first decision defines supported products, languages, and question types. Therefore, source checks can focus on the evidence those answers actually require.
Separate three uses of data
Training data helps fit model parameters. Its selection needs checks appropriate to that learning task, including separation from evaluation examples.
Retrieved sources provide evidence during an answer. Consequently, their currency, access permissions, and coverage matter even when the model itself remains unchanged.
Evaluation cases test behavior against expectations. Keep an independent set outside prompt tuning, and document any overlap with development examples.
Inspect a source register
The fictional collection contains four document groups. The table records observed problems rather than assigning a single readiness score.
A current date does not establish complete coverage. Likewise, a readable document does not establish permission to use it.
A useful register pairs each problem with an owner and next check. Missing evidence remains unknown until someone completes the required review.
| Source group | Observed condition | Next step |
|---|---|---|
| Current general returns policy | Approved version; effective date recorded | Confirm supported products |
| Archived returns policy | Old terms remain in search index | Exclude from current-policy retrieval |
| Custom-order exceptions | Only a scanned appendix; extraction omits its footnote | Verify extraction against original |
| Partner agreement | Use permissions unresolved | Hold outside the pilot collection |
Measure coverage against real questions
Our fictional task set contains twenty planned question types. Twelve concern standard returns, five concern custom orders, and three concern partner terms.
Verified source coverage currently supports only the twelve standard questions. Thus, twelve of twenty planned types have checked evidence, or sixty percent.
That coverage figure describes this selected task set, not expected accuracy or traffic volume. A hundred duplicate policy copies would not fill the eight gaps.
Repair meaning, not just file format
An extraction check compares the original with the text supplied to the system. Inspect headings, table relationships, exceptions, dates, and footnotes that change the answer.
The missing footnote says opened custom orders cannot use the standard return route. Without it, an otherwise readable extract changes the policy meaning.
The repair record keeps the source version, corrected extraction, reviewer, and affected cases. Then repeat the custom-order questions before expanding coverage.
Make readiness a bounded decision
A limited pilot could cover standard returns only, after its source and permission checks pass. Unsupported products must route to a human process.
A change owner maintains the collection when a policy changes or access is revoked. Also check derived indexes and cached answers within the application’s control.
A stop condition names an observation that invalidates the boundary. For example, a custom-order question receiving an unsupported standard approval should pause that response path.
Practice: defend the source decision
Your assignment classifies each source group as ready for this task, blocked, or needing evidence. Explain the condition behind every classification.
Your test plan includes a standard question, a custom-order exception, an archived-policy query, and a partner question. State the expected answer boundary first.
Your decision memo names the permitted pilot scope and unresolved gaps. Finally, explain why sixty percent coverage cannot become a sixty percent accuracy claim.
Compare with the worked answer
Worked answer: Use the checked standard-policy sources only within the defined pilot scope. Keep the unresolved partner agreement outside the collection. Remove archived policy from current-policy retrieval and repair the missing exception before adding custom orders. Unsupported questions should receive an explicit limitation and a human handoff. The 12/20 figure counts covered question types; it does not measure generated-answer correctness or represent the distribution of future requests.
Use the method on your own task
Editable worksheet: Download the Markdown template and open it in a text editor. Use fictional or approved information; no account or paid service is required.
Download the worksheet (.md) ↓What to carry forward
Create a source register, identify a coverage gap, and justify a bounded decision about using the data.
Show your reasoning with the exercise above. Keep the evidence, the decision, and the remaining uncertainty together.