1 / Evidence
Start from one observed failure with a clear boundary
Choose a run where the record shows the exact unwanted behavior, such as an illegal action, stale write, duplicate logical mutation, or missed stop condition.
- Save the game and version identity.
- Save the observation immediately before the failure.
- Save the attempted action and typed result.
- Redact credentials, private state, and unrelated identifiers.
2 / Fixture
Freeze the environment without freezing agent prose
Use deterministic inputs where the environment supports them, pin the configuration and protocol, and define the action-state boundary that matters.
The same environment does not guarantee identical model text or actions. Protect a concrete invariant such as legal tool selection, schema validity, one accepted effect, or stop behavior.
3 / Check
Write the smallest assertion that catches the failure
- Required read occurs before the mutation.
- Forbidden action is never accepted.
- Corrected payload uses a new idempotency key.
- A stale mutation produces no state change.
- Final state matches the authoritative record, not the final message.
4 / Gate
Rerun the baseline and candidate under one protocol
Show that the known-bad configuration still triggers the assertion and that the candidate no longer does, then retain both records with the test identity.
One repaired case proves only that bounded regression. Add broader coverage when another observed failure or accepted requirement requires it, not to manufacture a universal reliability claim.