Evaluation API reference

The supported evaluation surface is praval.eval. It is provider-neutral and available in the base wheel. PostgreSQL methods require the storage dependency; praval.eval.ragas is optional and intentionally not re-exported.

Datasets and immutable records

  • EvalCase, EvalSuite, LoadedEvalCase, LoadedEvalSuite, and load_jsonl_suite() define bounded datasets.

  • EvaluationRun, EvaluationSubject, MetricResult, JudgeResult, GateResult, and EvaluationResult are immutable persisted records.

  • EvaluationBaseline, EvaluationJob, and EvaluationAttempt cover explicit regression and durable online lifecycle.

  • Status/operator/aggregation enums make serialized decisions stable.

Validation errors are ordinary Pydantic ValueError failures. EvalDatasetError adds file/line context without exposing unrelated data.

Execution and judges

  • EvalRunner executes a LoadedEvalSuite with bounded concurrency.

  • EvaluationTarget and TargetResult define the async target seam.

  • AgentEvaluationTarget adapts a non-persistent ordinary Agent and enforces one observation per case.

  • Judge, JudgeContext, ModelJudge, and AgentJudge define strict evaluator composition.

  • EvaluationExecutionError, JudgeConfigurationError, and JudgeResponseError report invalid execution or configuration. Judge runtime failures normally become error results.

Metrics, gates, and plugins

  • Metric, ExactMatchMetric, TerminalSuccessMetric, and ToolCallMatchMetric are asynchronous provider-free metrics.

  • builtin_metrics(), discover_metric_plugins(), and available_metrics() expose deterministic plugin discovery under METRIC_ENTRY_POINT_GROUP.

  • MetricPluginError reports malformed, duplicate, or conflicting plugins.

  • Gate, evaluate_gate(), compare_evaluation_runs(), and promote_evaluation_baseline() implement explicit quality policy.

  • GateEvaluationError, MetricComparison, and RunComparison describe invalid or completed comparisons.

Stores

EvaluationStore is the async structural contract. Both SQLiteEvaluationStore and PostgresEvaluationStore provide migrations, immutable idempotent writes, queries, baselines, jobs, leases, retries, and attempts. Call migrate() before use and close() at shutdown. EvaluationStoreError is the base persistence error; EvaluationConflictError means an identity collision with different data.

Online evaluation

  • trace_sampled() makes a deterministic decision from a 128-bit trace ID.

  • OnlineEvaluationProcessor and OnlineContextLoader are async application seams.

  • OnlineSubjectEvaluator composes configured public judges and metrics.

  • OnlineEvaluationService.start(), .record(), .stats(), and .shutdown() own the bounded scheduler/worker lifecycle.

  • OnlineEvaluationStats contains counters and queue metadata only.

  • evaluation_call_scope() and is_evaluation_call() prevent recursive sampled evaluation.

For full signatures, parameter types, defaults, return values, and source docstrings, see the generated praval.eval modules under API Reference.