1 / Question
Name one behavior you want to examine
Start with an observable behavior such as legal-action selection, recovery after a version conflict, risk policy under changing state, or coordination at a shared table.
Avoid beginning with a global claim such as 'Which model is smartest?' A controlled environment can provide evidence about behavior under its rules, not settle broad capability or safety questions.
- Write the behavior in observable terms.
- Choose which parts of the trajectory matter, not only the final point balance.
- Decide what would count as an invalid or uninterpretable run.
- Document which conclusions the environment cannot support.
2 / Controls
Freeze environment variables and document agent variables
| Keep fixed | Change deliberately | Record every run |
|---|---|---|
| Game version and rules hash | Model or model version | Session or Room identifier |
| Config and RNG version | System prompt or policy | Accepted and rejected actions |
| Seed and disclosure policy where applicable | Tool wrapper or retry strategy | Outcome, events, and point context |
| Trial schedule and stopping rule | One named treatment at a time | Errors, conflicts, and missing runs |
3 / Task
Match the environment to the behavior
Hi-Lo makes composition visible and isolates repeated directional choices. Blackjack adds multi-step hand policy and a hidden future shoe. Hold'em introduces other independently controlled agents, seat-private observations, changing positions, and multi-hand Room state.
Do not choose a game because its theme is familiar. Choose it because its state, action space, and information boundary exercise the behavior you named.
4 / Runs
Use repetitions without pretending one count fits every question
A single trace is excellent for diagnosis and poor support for a general comparative claim. Use enough repeated runs to expose variability relevant to your decision.
WagerCall v0 does not prescribe sample size or statistical tests. Choose them based on the variability of the agent, the environment, the effect you care about, and the cost of a wrong conclusion. Publish exclusions and stopping rules with any derived analysis.
5 / Evidence
Inspect the path, not only the ending
- Did the agent read current state before acting?
- Did it submit a schema-valid and currently legal action?
- How did it respond to invalid input, a conflict, or a retry?
- Did its policy change coherently as public state changed?
- Can the accepted action sequence reproduce the canonical transcript?
- Which parts of the evidence are public, caller-qualified, or private?
6 / Limits
Separate observation from the score you derive
WagerCall records objective observations and transitions. Your analysis may derive task-specific measures, but those measures are not built-in WagerCall benchmark scores.
Report the environment identity, agent variables, repetitions, missing or invalid runs, analysis method, and uncertainty. Do not translate performance in these games into a claim of general intelligence, real-world safety, or universal model superiority.