Skip to content

Guide / Decisions under uncertainty

Evaluate AI agent decisions under uncertainty.

WagerCall environments reveal some state and deliberately withhold future outcomes or seat-private information. Evaluate whether an agent's actions follow a declared policy using only the information available at decision time, then report outcomes separately.

For
Evaluators studying sequential choices with incomplete information
Outcome
A decision-quality review grounded in time-appropriate observations

1 / Boundary

Define exactly what the agent may know at each decision

  • Public observation and legal actions returned by the current version
  • Caller-qualified private observation for an authorized Room seat
  • Prior public events in canonical order
  • No unrevealed shoe order, future outcome, other seat's private state, or hidden chain-of-thought

2 / Hypothesis

State a policy that maps observations to actions

Write the decision rule, tie-breaking behavior, point-sizing rule, and stop conditions before the run so each accepted action can be checked against the observation that preceded it.

If the evaluator asks the agent for a confidence estimate, store that evaluator-owned artifact beside the run identity. WagerCall does not record or infer hidden reasoning.

3 / Review

Judge the action from decision-time information

  1. 01

    Read the prior observation

    Use the exact public or caller-qualified state available before the action.

  2. 02

    Apply the declared policy

    Compute the action or acceptable action set without using the later outcome.

  3. 03

    Compare the attempt

    Record whether the submitted action matched the policy and was legal.

  4. 04

    Attach the outcome later

    Use settlement to describe consequence, not to rewrite whether the decision followed the rule.

4 / Limits

Report policy adherence, outcomes, and uncertainty separately

  • Eligible decisions and policy-adherent actions
  • Rejected attempts and recovery behavior
  • Outcome distribution under the exact environment versions
  • Run count, environment identities, and missing evidence
  • No claim that one bounded game proves general reasoning quality

Next step

Select a bounded decision environment

Inspect its information and action boundaries before designing the run.

Select a bounded decision environment