Eval Dataset Schema and Grading Contract¶

This page continues two nearby topics:

• Chapter 13. Offline Evals, Online Evals, and Regression Gates
• Trace Schema and Event Catalog
• Evidence Spine: From Request to Rollout Judgment

And connects them to the runnable package:

• Reference Package

If the trace schema page answers “how do we describe what happened inside a run?”, this page answers “how do we describe what we expect from the system as an eval artifact?”

Why an explicit eval dataset schema matters¶

Many teams say they “have evals”, but in practice that often means:

• a spreadsheet with a few manual examples;
• a set of unrelated prompt cases;
• JSON without a stable structure;
• a mix of ground truth, expectations, and reviewer comments in one field.

That is a problem for three reasons:

• comparisons between versions become blurry;
• regression gates are hard to automate;
• trace grading and dataset grading live in separate worlds.

That is why it helps to treat an eval dataset as a contract.

Eval integrity as a first-class control¶

OpenAI's SWE-Bench Pro audit shows why that contract is not enough by itself: even a realistic benchmark can produce noisy signal when the tasks are broken. OpenAI found that an automated pipeline flagged 200 of 731 public-split tasks as broken, while a campaign with five experienced engineers per task identified 249. In the language of this book, the eval artifact itself needs quality assurance because it shapes deployment safety, research priorities, and safety-case claims.

A minimal defect taxonomy is useful directly in the schema:

• overly_strict_tests: hidden tests require a specific implementation that the prompt did not require;
• underspecified_prompt: the prompt omits requirements that the oracle later enforces;
• low_coverage_tests: tests allow incomplete solutions to pass;
• misleading_prompt: the prompt points toward behavior that conflicts with tests or the gold patch.

That means verifier_outputs should sit beside a separate eval_audit_record. This record describes not how the agent performed on a scenario, but whether the measurement artifact is trustworthy: source_task_id, oracle_type, defect_labels, agent_audit_refs, human_reviewer_count, human_agreement, reviewer_confidence, and decision_impact.

The practical pattern is not "let an agent grade the eval." The stronger shape is agent-assisted eval audit + independent human adjudication: agents help inspect prompts, tests, traces, and patches at scale, while the final label, confidence, and release impact remain a separate human-reviewed artifact.

Minimal eval artifact shape¶

For agent systems, it is very useful for one dataset item to contain at least:

• scenario_id
• labels
• user_inputs
• expected_outcomes
• risk_class

A minimal example looks like this:

{
  "scenario_id": "support_ticket",
  "labels": ["write_path", "approval_required", "ticketing"],
  "user_inputs": [
    "Please create a ticket for this onboarding issue."
  ],
  "expected_outcomes": {
    "latest_status": "success",
    "approval_wait_runs": 1,
    "required_output_substrings": [
      "waiting for human approval"
    ]
  },
  "risk_class": "high"
}

That is already much more useful than “here is an example prompt”.

Why labels are not enough without expected outcomes¶

Labels help you group scenarios:

• retrieval
• approval
• memory
• safety
• multi-turn

But labels alone do not tell you what successful behavior actually means.

That is why an eval dataset should usually separate:

• labels as the scenario class;
• expected_outcomes as the desired result;
• grading_rules as the check logic;
• verifier_outputs as the structured grading result, including verifier identity and contract version.

What a grading contract is¶

A grading contract exists to remove ambiguity between “an example” and “a pass criterion”.

In practice, that means a scenario should explicitly define:

• which fields are evaluated;
• which check types apply;
• what counts as pass/fail;
• what may be treated as a warning versus a blocking failure.

A good grading contract answers:

“Would a different reviewer or pipeline reach the same conclusion on the same scenario tomorrow?”

Useful grading rule types¶

For reference-grade agent evals, it helps to distinguish at least these rules:

• status_equals
• contains_substring
• max_tool_calls
• approval_required
• policy_violation_absent
• memory_write_absent
• process_score_present
• outcome_score_present
• failure_attribution_valid
• failed_run_traceable
• sandbox_profile_review
• stop_condition_verified
• delegation_budget_respected
• single_vs_multi_agent_regression

failed_run_traceable becomes important once release review expects failed-run drills. It checks that a degraded path did not merely fail, but failed in a way the team can still inspect through status, a concrete failure reason such as failure_reason, trace linkage, and governed release identity.

sandbox_profile_review matters for sandbox-backed paths: it checks that workspace materialization, shell/filesystem permissions, network/secrets posture, and snapshot/resume policy were explicitly represented as reviewable evidence instead of remaining implicit runtime settings.

stop_condition_verified matters for agent-run paths where a free-text “done” is not enough. It checks that the scenario carries an explicit stop condition, verification mechanism, verification result, verifier actor, and evidence link such as test output, trace, screenshot, diff, or artifact.

delegation_budget_respected matters for manager/subagent paths. It checks that fanout passed an explicit gate, subagent_count stayed within limit, context_handoff_size and token_budget remained within scenario bounds, and delegation_reason explains why the single-agent path was insufficient.

single_vs_multi_agent_regression compares modes. It checks that multi-agent genuinely wins on read-heavy breadth-first work and fails or blocks on write-heavy shared-state work when conflicting actions, approvals, context loss, or merge_conflict_risk increase.

That means the grading contract should not focus only on the final answer text, but also on system behavior.

How this connects to traces¶

A useful practical model looks like this:

• the trace schema describes actual run behavior;
• the eval dataset schema describes expected behavior;
• the grading contract maps one to the other.

This is the point where observability stops being only a way to look backward and becomes part of release decisions.

What the reference runtime already supports¶

In agent_runtime_ref, this command:

.venv/bin/python -m agent_runtime_ref export-eval-dataset --output artifacts/eval-dataset.json

already produces a small structured artifact with:

• multiple session scenarios;
• labels;
• expected_outcomes;
• a failed-run drill scenario that preserves failed status and failure_reason in session export and eval expectations.

The bundled export contract is intentionally concrete. Session eval config validation also keeps malformed eval specs separate from failed eval results with Session eval specs must be a mapping, Session eval spec must be a mapping, Session eval spec key must be a string, Session eval spec key must not be empty, and Session eval spec keys must be unique.

The export contract is intentionally concrete: the default dataset_name is agent-runtime-ref-eval-seed; the top-level summary includes session_count, session_ids, run_count, failed_runs, traceable_failed_runs, trace_ids, failed_trace_ids, idempotency_keys, approval_ids, approval_capability_names, pending_approval_ids, pending_approval_capability_names, approval_status_counts, and latest_failure_reason; approval-backed scenarios also carry approval_status_counts in expected_outcomes. The built-in failed_run_timeout scenario proves a known pre-dispatch failure and traceability. The separate unknown_effect_reconciliation scenario produces side_effect_unknown, expects one reconciliation_runs entry, and owns the duplicate_ticket_eval_passed label, max_ticket_side_effects: 1, and blocking duplicate_ticket_guard rule. profile_memory uses memory_read, profile_lookup, and grounded_answer; mixed_session uses multi_run, approval_then_memory, session_evals, and required_run_count; support_ticket uses sandbox_profile_review and sandbox_profile_reviewed.

It is not yet a full industrial eval framework, but it is already a reasonable seed for:

• regression grading;
• scenario comparison;
• rollout review;
• manual expansion of the eval set.

What a production dataset schema should add¶

Once the system becomes more serious, it is useful to extend the dataset schema with verifier verdict record fields:

• dataset_version
• scenario_owner
• source_trace_ids
• grader_type
• blocking
• notes_for_review
• verifier_outputs
• failure_attribution
• verdict_id
• verifier_id
• verifier_contract_version
• input_refs
• verifier_evidence_refs
• blocking_decision
• comparison_baseline
• reviewer_override
• sandbox_profile_contract
• workspace_manifest_ref
• snapshot_policy
• stop_condition
• verification_command
• verification_result
• verifier_actor
• evidence_refs
• eval_audit_record
• oracle_type
• defect_labels
• agent_audit_refs
• human_reviewer_count
• human_agreement
• reviewer_confidence
• decision_impact

That is when the eval artifact starts behaving like part of release discipline, not just temporary JSON.

Verifier Verdict Acceptance Criteria¶

A verifier verdict is a contract, not a reviewer comment, only if it passes a few checks:

• it has stable verdict_id, verifier_id, and verifier_contract_version;
• inputs (input_refs) and evidence (verifier_evidence_refs or evidence_refs) point to traces, scenarios, and policy versions;
• process_score, outcome_score, and failure_attribution are separate, not collapsed into one label;
• blocking_decision, comparison_baseline, and reviewer_override explain whether release is blocked, warned, or allowed;
• stop_condition, verification_command, verification_result, and verifier_actor record how run completion was checked.

Example grading contract¶

Here is a workable skeleton for a failed-run drill scenario:

scenario_id: failed_run_timeout
labels:
  - failed_run
  - tool_timeout
  - failure_drill
grading_rules:
  - type: status_equals
    expected: failed
    blocking: true
  - type: contains_substring
    expected: tool_timeout
    blocking: true
  - type: failed_run_traceable
    expected: true
    blocking: true
  - type: sandbox_profile_review
    expected:
      sandbox_profile_contract: sandbox-profile-v1
      workspace_entries_reviewed: true
      permissions_profile: restricted-shell-network-denied
      network_secrets_posture: network:denied,secrets:none
      snapshot_policy: required_on_completion
    blocking: true
  - type: stop_condition_verified
    expected:
      stop_condition: no duplicate ticket side effect after timeout replay
      verification_command: .venv/bin/pytest tests/test_docs_surface.py
      verification_result: pass
      verifier_actor: deterministic_gate
      evidence_refs:
        - trace:trace_123
        - artifact:pytest-output
    blocking: true
verifier_outputs:
  verdict_id: verdict_failed_run_timeout_2026_05
  verifier_id: fara-process-review
  verifier_contract_version: verifier-v2
  input_refs:
    - scenario:failed_run_timeout
    - trace:trace_123
    - policy_bundle:policy-bundle-v3
  process_score: 0.92
  outcome_score: 0.35
  failure_attribution: uncontrollable_environment
  blocking_decision: warning_only
  comparison_baseline: release-2026-05-previous
  reviewer_override: none
  verifier_evidence_refs:
    - trace:trace_123
    - screenshot:step_7

The point is that the contract evaluates not only the final text, but also the correct operational shape of behavior, including whether the concrete failed condition remains visible enough for later review.

This becomes especially important for long-horizon agents, where a binary pass/fail verdict often hides the difference between correct behavior with a blocked outcome and unsafe behavior that happened to end in nominal success.

Why multi-run sessions matter¶

For agent systems, an eval item often needs to describe not one request, but a short related sequence.

For example:

1. the user asks to create a ticket;
2. then asks what the agent remembers about preferences;
3. then asks for the next step.

If the dataset cannot describe such a sequence, you can test single-turn behavior fairly well, but session behavior much less well.

That is why session exports and eval dataset exports should be designed together.

What not to do¶

Several mistakes are very common:

• mixing scenario metadata and grading logic in one text field;
• keeping only happy-path cases;
• not declaring expected outcomes explicitly;
• grading only the final answer and ignoring policy or tool behavior;
• not versioning the dataset;
• not auditing the quality of tasks, oracles, and hidden tests;
• not linking dataset items to trace evidence or incident history;
• collapsing verifier output into a single weak verdict with no process/outcome split or failure attribution;
• requiring sandbox_profile_review in rollout, but having no grading rule that checks workspace, permissions, and snapshot/resume evidence.
• letting the agent finish a task without stop_condition_verified and without evidence that can be checked after the session.

That makes eval culture fragile.

What to Do Right Away¶

Start with this short list and mark every "no" explicitly:

• Does every scenario have a stable scenario_id?
• Are labels separate from expected outcomes?
• Do you have grading rules, not just reviewer prose?
• Can you evaluate behavior, not only text?
• Do you have an eval_audit_record that captures defect labels, oracle type, reviewer confidence, and decision impact?
• Can the verifier output separate process_score, outcome_score, and failure_attribution?
• Can you tell which verifier identity and contract version produced that grading output?
• Is there a dedicated rule for sandbox-backed paths that checks sandbox profile contract, workspace entries, permissions, and snapshot/resume evidence?
• Is there a rule that checks stop condition, verification command/result, verifier actor, and evidence refs before run completion is accepted?
• Do you support multi-run sessions?
• Do you have dataset versioning and ownership?

If several answers are “no,” you probably have examples, but not yet a proper eval dataset schema.

What to Do Next¶

• Trace Schema and Event Catalog
• Policy Bundle Schema and Approval Contract
• Lifecycle Artifact Schema
• Reference Package
• Chapter 13. Offline Evals, Online Evals, and Regression Gates