Install and five-minute quickstart

Install

Core evaluation contracts, deterministic metrics, SQLite storage, runners, judges, gates, and the CLI are in the base package:

python -m pip install praval

Install optional integrations only when needed:

python -m pip install "praval[eval-ragas]"  # RAGAS 0.4 adapter
python -m pip install "praval[storage]"     # shared PostgreSQL deployments

Define a dataset

Create evals/answers.jsonl:

{"id":"capital-france","input":"Capital of France?","expected_output":"Paris","tags":["smoke"]}

The loader keeps the input and expected output only in the active LoadedEvalCase. Persisted EvalCase records contain content references.

Run one deterministic suite

import asyncio
from datetime import datetime, timedelta, timezone

from praval.eval import (
    EvalRunner,
    ExactMatchMetric,
    Gate,
    SQLiteEvaluationStore,
    TargetResult,
    evaluate_gate,
    load_jsonl_suite,
)
from praval.models import ExecutionObservation, ObservationKind, ObservationStatus


class Target:
    async def evaluate(self, case):
        now = datetime.now(timezone.utc)
        return TargetResult(
            observation=ExecutionObservation(
                observation_id=f"observation-{case.case.case_id}",
                run_id=f"run-{case.case.case_id}",
                kind=ObservationKind.AGENT,
                agent_name="answerer",
                started_at=now,
                ended_at=now + timedelta(milliseconds=1),
                duration_ms=1,
                status=ObservationStatus.OK,
            ),
            output="Paris",
        )


async def main():
    gate = Gate(
        gate_id="exact-pass-rate",
        metric="exact_match",
        aggregation="pass_rate",
        operator=">=",
        threshold=1.0,
    )
    suite = load_jsonl_suite(
        "evals/answers.jsonl",
        suite_id="answers-v1",
        name="Answer quality",
        target="agent:answerer",
        metrics=("exact_match",),
        gates=(gate,),
    )
    store = SQLiteEvaluationStore(".praval/evaluations.db")
    await store.migrate()
    result = await EvalRunner(
        store=store,
        target=Target(),
        judges={},
        metrics={"exact_match": ExactMatchMetric()},
    ).run(suite, evaluation_run_id="local-smoke-1")
    metric_results = await store.list_metric_results(
        evaluation_run_id=result.evaluation_run_id
    )
    decision = evaluate_gate(gate, metric_results)
    print(result.status.value, decision.status.value)
    await store.close()


asyncio.run(main())

Expected output is completed passed. Reusing the same run ID with different immutable data raises EvaluationConflictError; choose a new ID for a new run.

CLI configuration

For a registered Agent, praval eval run builds the target, metrics, judges, and gates from praval.toml:

schema_version = 1

[eval]
enabled = true
store = "sqlite"
offline_concurrency = 4

[eval.stores.sqlite]
path = ".praval/evaluations.db"

[eval.suites.answers]
dataset = "evals/answers.jsonl"
target = "agent:answerer"
metrics = ["exact_match"]

[[eval.suites.answers.gates]]
gate_id = "exact-pass-rate"
metric = "exact_match"
aggregation = "pass_rate"
operator = ">="
threshold = 1.0

Register the target agent from a module, then run:

praval eval run answers --module myapp.agents --json

Exit codes are 0 for success, 1 for a failed gate or regression, and 2 for configuration, dataset, store, target, judge, or execution errors. Continue with Recommended agent and evaluator-agent patterns, then Gates, baselines, and CI before adding a paid judge.