Short answer
Evaluate the action from the observation, not the hidden state
The relevant question is whether the action was legal and consistent with the declared policy given what the agent could know at that sequence point.
A later reveal can make one alternative look obviously better. That does not mean the earlier choice ignored visible evidence. Using post-outcome information to label decision quality turns stochastic resolution into hindsight.
The evaluator may use hidden state to check environment integrity or calculate an explicitly omniscient counterfactual. It should not present that result as the same measure as decision quality under the agent's actual observation.
Evidence scopes
Separate decision-time, outcome-time, and privileged information
| Information set | Examples | Appropriate use |
|---|---|---|
| Decision-time | Public state, own private state, legal actions | Judge the agent's choice |
| Outcome-time | Later cards, settlement, terminal state | Judge the realized result |
| Privileged environment | Hidden future order, other seats' private state | Integrity checks or labeled counterfactuals |
| Agent configuration | Prompt, policy, model, tool wrapper | Explain study conditions, not hidden game state |
Evaluation rule
A declared policy makes decision quality testable
Before the run, define the observable conditions that should change an action and the invariants that should hold across equivalent decisions.
The policy can be coarse. It might require legal actions, bounded synthetic-point exposure, or a consistent response to the same visible composition class. What matters is that the rule uses fields available to the agent and does not change after the outcome is known.
Without that reference, a reviewer can still describe behavior and outcomes but should be cautious about calling a choice optimal, rational, or erroneous.
Product example
WagerCall separates observations from authoritative outcomes
Session and Room records can connect caller-visible state, attempted actions, transitions, and later settlement while preserving private-information boundaries.
- Use the pre-action observation and legal-action set for policy review.
- Use the terminal state and point entries for realized outcome reporting.
- Treat hidden future randomness and other owners' private state as outside the agent's evidence set.
- Report repeated decisions because one stochastic ending is not a stable policy measure.