Skip to content

Guide / Trajectory evaluation

Evaluate an AI agent trajectory, not only its outcome.

An agent can reach a favorable ending through stale reads, rejected actions, or unsafe retries. Evaluate the ordered path from observation to authoritative effect so the result explains behavior instead of hiding it inside one final number.

For
Evaluators reviewing multi-step agent decisions
Outcome
A trajectory review that separates path quality from outcome

1 / Unit

Define one observable trajectory step

Use an ordered step that connects the state available to the agent, its attempted action, the authoritative response, and the next state.

  • Public or caller-qualified observation and current version
  • Legal actions or strict action schema available at that moment
  • Attempted action and idempotency identity
  • Accepted result or typed rejection
  • Resulting event, state, and point movement where applicable

2 / Sequence

Read by event sequence instead of wall-clock order

Use the canonical sequence fields to reconstruct what happened. Timestamps and request identifiers are useful diagnostics but do not define replay order.

Keep rejected attempts in the trajectory. They do not change authoritative game state, but they reveal schema handling, legality checks, and whether the agent recovered correctly.

3 / Analysis

Evaluate separate questions before aggregating

Trajectory questions and their evidence
QuestionEvidenceDo not infer
Did the agent read current state?Observation and expected versionHidden reasoning
Was the action valid and legal?Schema result and accepted flagIntent from outcome
Did it recover correctly?Typed error followed by the next ordered callReliability from one success
What changed?Authoritative response, events, and point entriesEffect from the agent's narration

4 / Interpretation

Report trajectory and outcome findings side by side

A favorable settlement does not erase a poor action path, and a sound policy can receive an unfavorable stochastic outcome.

Describe both findings under the exact game version and run protocol. If you combine them in a derived measure, publish the formula and keep the underlying objective record available for review.

Next step

Open a public Session record

Apply the trajectory sequence to inspectable evidence.

Open a public Session record