Cases, datasets, and suites

Evaluation datasets are UTF-8 JSON Lines. Each non-empty line is one bounded JSON object.

{"id":"case-001","name":"Known answer","input":{"question":"2+2"},"expected_output":{"answer":4},"reference_contexts":["Arithmetic over integers"],"expected_tool_calls":["calculator"],"tags":["smoke","math"],"metadata":{"owner":"quality"}}

Field	Required	Contract
id	yes	non-empty and unique in the file
input	yes	finite JSON, bounded by loader limits
name	no	defaults to id
expected_output	no	finite JSON; required by some metrics
reference_contexts	no	JSON array; required by some RAGAS metrics
expected_tool_calls	no	ordered array of tool names
tags	no	array used for deterministic selection
metadata	no	bounded scalar values only

load_jsonl_suite() rejects duplicate IDs, non-finite numbers, oversized lines/content, unknown requested IDs, and empty selections. Selection is stable: explicit case IDs are sorted; tag filters are deterministic; a seeded limit hashes case IDs before selecting and then restores stable order.

Persisted EvalCase values hold ContentReference objects with SHA-256, size, media type, and a dataset URI. The loaded input, expected output, and reference contexts remain ephemeral. Dataset storage and access control remain an application responsibility.

A suite pins target, case IDs, judge names, metric names, gates, and tags. Version the suite ID when its semantic contract changes. EvalRunner parallelizes cases up to concurrency, but each AgentEvaluationTarget also protects its ordinary agent’s mutable history. Use separate agent instances when true target parallelism is required.