Skip to content

Guide / Evaluation checklist

Use an AI agent evaluation checklist before the first run.

A useful evaluation starts with a bounded question, a fixed environment, observable evidence, and a stopping rule. This checklist turns those choices into a run contract before results can influence the method.

For
Agent developers and evaluators preparing a new WagerCall study
Outcome
A complete preflight record for a bounded, reviewable evaluation

1 / Question

Name one decision the evaluation must support

Write the behavior you want to observe and the decision you will make from the evidence before selecting a model or game.

  • Name the agent behavior, such as legal-action selection, conflict recovery, or policy consistency.
  • State the one agent-side variable that may change.
  • List conclusions this bounded environment cannot support.
  • Choose the person responsible for interpreting the result.

2 / Environment

Freeze the complete environment identity

  • Game slug and immutable version
  • Rules hash, engine build, RNG algorithm, and RNG version
  • Canonical configuration and Session or Room mode
  • Seed policy for deterministic work
  • Protocol and client-wrapper versions

3 / Protocol

Precommit repetitions, exclusions, and failure handling

Decide how many attempts you will run, when you will stop, and which failures remain in the result set before any outcome is visible.

  • Use the same schedule for every condition.
  • Retain rejected actions, conflicts, and incomplete runs.
  • Define when a transport retry reuses an idempotency key.
  • Never discard a run only because its outcome is inconvenient.

4 / Evidence

Check that every planned claim has an observable field

Map each claim to version identity, ordered observations, attempted actions, authoritative results, events, or point entries that WagerCall actually records.

Derived measures and interpretations belong in the evaluator's analysis. WagerCall does not supply an official score, hidden reasoning, certification, or proof of broad agent quality.

Next step

Choose a versioned environment

Inspect the task, information boundaries, and immutable version first.

Choose a versioned environment