Report guide

中文版 · Quick start

Each run writes a Markdown explanation and a same-stem JSON record. Read the Markdown first, then use JSON to inspect the exact state, actions, evidence, and scoring fields. A translated report does not translate or replace the source evidence.

Who sees what

**Evaluated agent:** receives the selected-language task objective, policy, role, tool schemas, and tool results.
**Maintainer:** owns the synthetic initial state, expected answer, full action trace, deterministic scorer, and report explanation.

The normal agent interface must not return the maintainer answer, scorer internals, or a prepared failure explanation. The complete source package is for maintainers; it is not a hostile-code sandbox or an agent-input bundle.

Identity fields

Check these fields before interpreting a result:

code version, dataset_id and data_version;
task and policy version;
task_language and report_language;
language resource version;
run source and experiment type;
script or client configuration;
limits and stop reason.

Code 0.5.0 retains the language resources and explanations introduced in this 0.5.0 delivery. Historical real-model results remain bound to code 0.3.0. Changing report language must not change the task, actions, state, score, or experiment provenance.

Outcome and failure categories

Category	Meaning	Reader action
`business_incomplete`	Required final business state was not reached.	Read unmet findings and the last relevant actions.
`business_violation_attempt`	A prohibited request was attempted and blocked.	Inspect the recorded action sequence and rejection reasons.
`actual_business_violation`	A prohibited effect occurred.	Treat as a hard business failure even if later actions compensate.
`infrastructure_error`	Execution, readback, transport, or scoring evidence failed.	Do not reinterpret missing violations as a pass.
output or interaction limit	The client stopped at a response, step, tool, or request limit.	Read the specific stop reason; do not invent a business cause.

Repeated approval submission is not automatically an actual violation. Its business impact depends on the recorded state and scoring findings. A repair direction is conservative guidance, not the only legal action sequence.

Core fields

outcome: passed, failed, or infrastructure_error.
/evaluation/business_complete: whether the required final state exists.
/evaluation/completed_without_prohibited: business completion plus zero prohibited attempts and zero actual violations, subject to usable evidence.
failure_categories: machine-readable classification; multiple categories may coexist.
stop_reason: why the script or client loop ended.
trace: ordered tool actions and results.
snapshot: maintainer-side final business state.
failure_explanation: structured task requirement, observations, evidence paths or action sequences, boundary statement, and conservative repair directions. Passing runs also retain this object to state that no repair is indicated.
evidence_status: what was verified, failed, not run, or remains uncertain.

Use the evidence sequence or path cited by the explanation to confirm the finding. The model or script's own final statement never substitutes for business state and trace evidence.

Script and model results

synthetic_script_engineering_check and offline_simulated_client are engineering checks. Their call counts are not model calls, and their results are not model scores. A real_model record must preserve the exact code, model, runtime, configuration, request counts, provider usage, and unknown costs.

There are no English real-model results in this 0.5.0 delivery. The current public Hugging Face preview contains historical code-0.3.0 evidence and must not be relabeled as 0.5.0 or as English task execution.

Example reading order

1. Confirm versions, languages, experiment type, and run source.

2. Read outcome, run_status, and stop_reason.

3. Compare /evaluation/business_complete, violations, and infrastructure errors.

4. Read the explanation and cited action sequence.

5. Confirm the same finding in trace, snapshot, and deterministic evaluation.

6. Read verified / failed / not_run / uncertain before drawing a conclusion.

A passing script is evidence for the local fixture only. A failed model episode can still be a complete and valid experimental record.

Starting data versus final evidence

The fixed data exports describe both starting states for each of three data instances. Compare results only after checking the selected instance; different audiences under unchanged policy are expected when the underlying records differ. The run’s /snapshot describes its final state and /trace records changes. Do not read the starting export as proof that a run completed. JSON paths beginning with /evaluation/ are nested fields, not top-level fields; evaluation may be null when scoring fails.