1 / Layers
Separate agent identity from environment identity
| Layer | Record | Why it matters |
|---|---|---|
| Agent | Model, settings, prompt, policy, wrapper build | Identifies the variable under evaluation |
| Protocol | MCP protocol and client implementation | Identifies tool-contract behavior |
| Environment | Game, version, rules hash, engine build, RNG identity | Identifies immutable task behavior |
| Run | Mode, canonical config, seed policy, attempt index | Identifies this execution condition |
2 / Capture
Capture the manifest before the first action
Persist or export the manifest when the Session or Room opens, before an outcome can influence which fields you remember to record.
- Use stable identifiers or digests for prompt and wrapper artifacts.
- Store configuration in a canonical serialized form.
- Record the exact game resource kind and version.
- Keep secrets, bearer credentials, and private observations out of the manifest.
3 / Compare
Reject accidental comparisons when manifests differ
Diff manifests before combining runs. A changed version, configuration, tool contract, or wrapper may explain a result that would otherwise be attributed to the model or prompt.
A game version change and an MCP protocol change are different axes. Preserve both rather than collapsing them into one application release label.
4 / Share
Publish enough identity to review the claim
- The variable changed between conditions
- Every environment field held constant
- Run count and exclusion policy
- Canonical record or replay references where sharing is appropriate
- Known limits and unavailable artifacts