# Report guide

[中文版](report-guide.zh-CN.md) · [Quick start](quickstart.en.md)

Each run writes a Markdown explanation and a same-stem JSON record. Read the Markdown first, then use JSON to inspect the exact state, actions, evidence, and scoring fields. A translated report does not translate or replace the source evidence.

## Who sees what

- **Evaluated agent:** receives the selected-language task objective, policy, role, tool schemas, and tool results.
- **Maintainer:** owns the synthetic initial state, expected answer, full action trace, deterministic scorer, and report explanation.

The normal agent interface must not return the maintainer answer, scorer internals, or a prepared failure explanation. The complete source package is for maintainers; it is not a hostile-code sandbox or an agent-input bundle.

## Identity fields

Check these fields before interpreting a result:

- code version, `dataset_id` and `data_version`;
- task and policy version;
- `task_language` and `report_language`;
- language resource version;
- run source and experiment type;
- script or client configuration;
- limits and stop reason.

Code 0.5.0 retains the language resources and explanations introduced in this 0.5.0 delivery. Historical real-model results remain bound to code 0.3.0. Changing report language must not change the task, actions, state, score, or experiment provenance.

## Outcome and failure categories

| Category | Meaning | Reader action |
|---|---|---|
| `business_incomplete` | Required final business state was not reached. | Read unmet findings and the last relevant actions. |
| `business_violation_attempt` | A prohibited request was attempted and blocked. | Inspect the recorded action sequence and rejection reasons. |
| `actual_business_violation` | A prohibited effect occurred. | Treat as a hard business failure even if later actions compensate. |
| `infrastructure_error` | Execution, readback, transport, or scoring evidence failed. | Do not reinterpret missing violations as a pass. |
| output or interaction limit | The client stopped at a response, step, tool, or request limit. | Read the specific stop reason; do not invent a business cause. |

Repeated approval submission is not automatically an actual violation. Its business impact depends on the recorded state and scoring findings. A repair direction is conservative guidance, not the only legal action sequence.

## Core fields

- `outcome`: `passed`, `failed`, or `infrastructure_error`.
- `/evaluation/business_complete`: whether the required final state exists.
- `/evaluation/completed_without_prohibited`: business completion plus zero prohibited attempts and zero actual violations, subject to usable evidence.
- `failure_categories`: machine-readable classification; multiple categories may coexist.
- `stop_reason`: why the script or client loop ended.
- `trace`: ordered tool actions and results.
- `snapshot`: maintainer-side final business state.
- `failure_explanation`: structured task requirement, observations, evidence paths or action sequences, boundary statement, and conservative repair directions. Passing runs also retain this object to state that no repair is indicated.
- `evidence_status`: what was verified, failed, not run, or remains uncertain.

Use the evidence sequence or path cited by the explanation to confirm the finding. The model or script's own final statement never substitutes for business state and trace evidence.

## Script and model results

`synthetic_script_engineering_check` and `offline_simulated_client` are engineering checks. Their call counts are not model calls, and their results are not model scores. A `real_model` record must preserve the exact code, model, runtime, configuration, request counts, provider usage, and unknown costs.

There are no English real-model results in this 0.5.0 delivery. The current public Hugging Face preview contains historical code-0.3.0 evidence and must not be relabeled as 0.5.0 or as English task execution.

## Example reading order

1. Confirm versions, languages, experiment type, and run source.
2. Read `outcome`, `run_status`, and `stop_reason`.
3. Compare `/evaluation/business_complete`, violations, and infrastructure errors.
4. Read the explanation and cited action sequence.
5. Confirm the same finding in `trace`, `snapshot`, and deterministic evaluation.
6. Read `verified / failed / not_run / uncertain` before drawing a conclusion.

A passing script is evidence for the local fixture only. A failed model episode can still be a complete and valid experimental record.

## Starting data versus final evidence

The [fixed data exports](data-guide.en.md) describe both starting states for each of three data instances. Compare results only after checking the selected instance; different audiences under unchanged policy are expected when the underlying records differ. The run’s `/snapshot` describes its final state and `/trace` records changes. Do not read the starting export as proof that a run completed. JSON paths beginning with `/evaluation/` are nested fields, not top-level fields; evaluation may be null when scoring fails.