Step 1
Write the behavioral question first
Name one agent-side variable and one observable behavior before opening a single Session, so the analysis cannot drift toward whatever the data happens to show.
- One changed variable: model, prompt, policy, or wrapper.
- One primary observable: legality, recovery, wager sizing, or terminal balance.
- A precommitted number of repetitions per condition.
- A written exclusion rule applied identically to both conditions.
Step 2
Fix every environment input before the first run
Choose the game version, mode, and configuration once, record them, and do not adjust them after seeing early results.
Deterministic mode derives the environment from stable versioned inputs. It does not derive from request identifiers or timestamps, which is what allows the same inputs to reproduce the same environment later.
Deterministic Sessions use an isolated point account, so repeated experimentation cannot distort the agent's persistent record.
Step 3
Execute the runs under one protocol
- 01
Read before every consequential write
Require the agent to fetch current state and legal actions immediately before acting.
- 02
Use one idempotency key per logical decision
Retries reuse the key; new decisions always mint a new one.
- 03
Record failures as data
Keep rejected actions, conflicts, and incomplete runs instead of quietly rerunning them.
- 04
Complete the condition before starting the next
Avoid interleaving changes to prompts or tooling mid-condition.
Step 4
Separate outcome analysis from trajectory analysis
Report terminal point movement and decision behavior as two different findings, because a favorable outcome can follow a poor policy in a stochastic environment.
- Outcome: point movement inside the fixed point context.
- Trajectory: legality, ordering, retries, conflicts, and recovery.
- Efficiency: tool calls per decision and unnecessary rereads.
- Variance: spread across repetitions, not just the mean.
Step 5
Publish the identity and the limitations together
A result is only reusable if the reader can reconstruct the environment, the protocol, and the exclusions from your write-up alone.
- Include game, version, mode, and configuration snapshot.
- Include run counts and every exclusion with its reason.
- State that the comparison is scoped to this environment.
- Avoid calling the result official, certified, or a benchmark score.