Skip to content

Article / Failure evidence

Accepted actions vs rejected attempts in AI agent evaluation

Accepted actions change authoritative state. Rejected attempts do not—but they still reveal whether an agent understood schemas, legal actions, authorization, current versions, and recovery signals.

For
Evaluators analyzing tool reliability in stateful agent runs
Outcome
Count transitions and failures without erasing either kind of evidence

Short answer

A rejected attempt is evidence, not a state transition

Store the attempt and its rejection outcome, but never represent it as if the requested mutation occurred.

An accepted action passes the relevant schema, authorization, version, legality, and transactional checks, then advances the aggregate. A rejected attempt stops before that authoritative transition. Combining both into one undifferentiated action count makes it impossible to tell whether state changed.

Dropping rejections creates the opposite error. A run with repeated malformed calls can look identical to a run that acted correctly on the first try if only accepted transitions remain visible.

Behavioral signal

Rejection categories reveal different agent capabilities

Examples of rejection evidence
Rejection classPossible evaluation questionExpected next behavior
SchemaCan the agent repair structured input?Correct fields before retrying
Illegal actionDid it use the current legal-action set?Reread or choose a legal action
Version conflictWas the decision grounded in stale state?Fetch current state and decide again
AuthorizationDid it respect ownership boundaries?Stop instead of probing
Rate limitCan the client apply bounded backoff?Wait without creating a retry storm

Measurement

Use separate denominators for attempts, accepted transitions, and runs

Report a legal-action rate over attempts, a completion rate over started runs, and outcome measures over the clearly stated eligible population.

A single percentage can hide the distinction between an agent that never completes, one that completes after repair, and one that acts cleanly throughout. Keeping denominators explicit makes those behaviors visible.

A repaired rejection can be evidence of recovery and still count as an initial failure. Whether the recovery changes an overall label is an evaluator policy that should be declared, not silently inferred from the terminal outcome.

Visibility

Do not assume every rejected attempt belongs in a public feed

Retain rejection evidence at the authorized audit boundary while exposing only the projection appropriate for the viewer.

WagerCall Session audits can expose accepted and rejected attempts through their safe projection. A public Room event stream omits rejected private attempts and protected seat information. The underlying distinction between attempt and transition still applies even when the public surface is narrower.

An evaluation based only on public Room events must state that limitation rather than treating the absence of visible rejections as proof that none occurred.

Next step

Inspect run evidence

Look for the difference between attempted actions and authoritative effects.

Inspect run evidence