Evaluation configuration reference

Evaluation uses typed praval.toml configuration. Defaults apply first, then the discovered/explicit file, supported environment variables, and explicit API construction. Unknown fields, bad references, and out-of-bound values fail before execution.

The EvalConfig fields are enabled, store, offline_concurrency, online, stores, ragas, judges, and suites, serialized beneath eval.

Core and stores

Field	Type	Default	Validation
eval.enabled	bool	false	no runtime work when false
eval.store	sqlite or postgres	sqlite	PostgreSQL block required when selected
eval.offline_concurrency	int	4	greater than zero
eval.stores.sqlite.path	string	.praval/evaluations.db	local/CI path
eval.stores.postgres.dsn_env	string	none	uppercase environment-variable name

The named DSN environment value is read at runtime and never included in diagnostics. Online evaluation requires store="postgres".

Judge profiles

Each [eval.judges.<name>] sets exactly one of agent or model.

Field	Default	Validation
timeout_seconds	60	positive
max_attempts	2	positive
allow_self_evaluation	false	explicit opt-in
allowed_tools	[]	subset of configured evaluator tools
tool_policy	evaluation_safe	or read_only
allow_side_effects	false	true rejected in v0.8.3
hitl_mode	suspend	or fail
max_input_tokens	16000	positive
max_cost_usd	0.25	positive finite
rubric	default outcome rubric	1 to 16,384 characters
rubric_version	1	1 to 128 characters
judge_version	1	1 to 128 characters

Agent references must exist under [agents]; model references must exist under [models]. Agent profile max_tool_rounds defaults to 8, and memory requires a stable memory_namespace.

Suites and gates

Each [eval.suites.<name>] requires dataset and target; judges, metrics, and gates default empty. Referenced judges and gate result names must exist. Each gate has optional gate_id, required metric, aggregation, operator, and finite threshold. required defaults true. percentile is required only for percentile aggregation and must be in (0, 100]. baseline_max_regression, when supplied, is finite and non-negative.

RAGAS

eval.ragas is optional. model and embedding reference named Praval profiles, timeout_seconds defaults to 60 and is positive, and strict_tool_order defaults true. Selected metrics may require only the model, only embeddings, or both; see Metrics, plugins, and RAGAS.

Online workers

Field	Default	Hard bound
enabled	false	boolean
sample_ratio	0.01	0 through 1
queue_capacity	1000	1 through 100,000
workers	2	1 through 64
max_attempts	3	1 through 3
max_enqueue_attempts	3	1 through 3
lease_seconds	180	positive, at most 3,600
job_timeout_seconds	120	positive, at most 1,800
poll_interval_seconds	0.1	positive, at most 60
retry_backoff_seconds	0.25	0 through 300
shutdown_timeout_seconds	5	positive, at most 300
max_subject_bytes	262,144	1 through 1,048,576

The lease must be longer than the processor timeout. Configuration never starts a worker; application lifecycle must explicitly construct and start the service.