Answer first
Start from every eligible run, then classify what happened
If a run entered the declared protocol, keep it in the primary denominator unless a predeclared eligibility rule proves that the environment—not the agent—made the run unusable.
A timeout, malformed call, repeated conflict, or voluntary stop can be the outcome of interest. Deleting it after the fact changes the question from ‘How often did this agent complete?’ to ‘How often did it complete among runs that completed enough for us to score?’
That narrower statistic may still be useful, but it needs its own name and denominator. It should not replace the all-eligible-run result.
Run accounting
Separate agent failures, environment failures, and missing evidence
| Classification | Example | Primary denominator |
|---|---|---|
| Completed | Reached a terminal authoritative state | Include |
| Agent failure | Invalid calls, abandonment, retry exhaustion | Include |
| Protocol stop | Declared action or time budget reached | Include and label |
| Environment failure | Service defect prevents a valid run | Report separately; apply predeclared rule |
| Missing evidence | Record cannot establish what happened | Disclose; do not silently drop |
Metrics
Different measures need explicit, defensible denominators
Completion rate can use eligible runs, legal-action rate can use attempted actions, and policy adherence can use decisions with sufficient decision-time evidence.
Those populations are not interchangeable. A run with no accepted actions still belongs in a completion denominator, even though it contributes no accepted transition to an outcome distribution. A rejected attempt belongs in the attempted-action denominator even when the agent later repairs it.
Publish the numerator and denominator beside every rate. Counts make exclusions and small samples visible in a way that a rounded percentage cannot.
Integrity
Declare exclusion rules before seeing which variant benefits
A fair comparison applies the same failure and exclusion policy to every model, prompt, or strategy under test.
- Name the eligible run population before execution.
- Record every started run and its terminal classification.
- Publish environment failures rather than hiding them inside agent results.
- Show sensitivity when a reasonable alternative exclusion rule changes the conclusion.