Evaluation

Praval evaluation runs deterministic metrics and versioned model or agent judges against immutable execution subjects. A subject contains one completed ExecutionObservation for an agent or workflow, with bounded aggregated facts for model calls, tools, retries, HITL, and Reef handoffs. Complete records live in an EvaluationStore; OpenTelemetry receives correlated summaries and trends, not the authoritative dataset.

Evaluation does not query Jaeger, Tempo, or another tracing backend. The target runtime hands the same provider-neutral observation independently to the configured observation recorder and to the evaluation runner. This preserves the dependency boundary: praval.observability never imports praval.eval.

• Install and five-minute quickstart
  • Install
  • Define a dataset
  • Run one deterministic suite
  • CLI configuration
• Evaluation configuration reference
  • Core and stores
  • Judge profiles
  • Suites and gates
  • RAGAS
  • Online workers
• Recommended agent and evaluator-agent patterns
  • Design an agent for evaluation
  • Pair target and evaluator
  • Least-privilege evaluator
  • Decision table
  • Anti-patterns
• Defining evaluator agents
  • Configuration
  • Capability ownership
  • Errors
• Evaluator flow and capability policy
  • Tools and side effects
  • Isolation and budgets
• Direct model judges
• Cases, datasets, and suites
• Agent and workflow evaluation
  • Single agent
  • Correlated workflow
• Metrics, plugins, and RAGAS
  • Built-in deterministic metrics
  • RAGAS
• Gates, baselines, and CI
• Sampled online evaluation
• Stores and retention
• Evaluation telemetry and trace correlation
• Cost, privacy, and security
  • Safe defaults
• Evaluation production recipes
  • 1. Local deterministic evaluation
  • 2. Paired target and evaluator agents
  • 3. Workflow judging
  • 4. CI regression gate and explicit baseline
  • 5. PostgreSQL shared store
  • 6. RAGAS through Praval runtimes
  • 7. Sampled online evaluation
  • 8. Metadata-only and redacted-content deployment
  • 9. Failure and graceful-shutdown exercise
  • 10. Correlate an evaluation result with its trace
• Troubleshooting
  • Target agent is not registered
  • Unknown judge or metric
  • Invalid judge output
  • RAGAS metric reports missing input
  • Gate failed
  • Store conflict or outage
  • Online jobs retry or dead-letter
  • Shutdown hangs
• Evaluation API reference
  • Datasets and immutable records
  • Execution and judges
  • Metrics, gates, and plugins
  • Stores
  • Online evaluation

Choose an evaluation mechanism

Question	Mechanism	Model call
Did execution terminate successfully?	TerminalSuccessMetric	No
Does structured output exactly match the reference?	ExactMatchMetric	No
Were the expected tools selected in order?	ToolCallMatchMetric	No
Does a custom deterministic rule apply?	praval.eval.metrics plugin	Usually no
Does semantic quality require a rubric?	ModelJudge	Yes
Does evaluation require safe tools or retrieval?	AgentJudge	Yes
Is the task RAG or agentic quality?	Optional RAGAS metric	Often

Start with deterministic checks. Add model judgment only for qualities that cannot be asserted reliably, and use an evaluator agent only when the judge genuinely needs its own tools, retrieval, memory, or HITL policy.

Record boundary

case -> target agent/workflow -> one immutable ExecutionObservation
                              -> EvaluationSubject
                              -> metrics and judges
                              -> gates and terminal EvaluationResult
                              -> EvaluationStore (authoritative)
                              -> linked OTel summaries (operational)

Prompts, responses, reference contexts, and judge evidence remain ephemeral or content-addressed by default. Metadata-only records carry identities, hashes, sizes, status, usage, cost, and bounded error types.

Executed evidence

The examples under examples/evaluation/ are credential-free and use only public APIs. Their positive, boundary, failure, and safety behavior is covered by tests/eval/, and the release suite runs them from the exact built wheel. PostgreSQL, Collector, RAGAS, RabbitMQ, privacy, performance, and shutdown contracts are separate explicit release gates.

Tutorial/recipe	Guide or executable	Executed evidence
Local deterministic suite	Quickstart	examples/evaluation/000_quickstart.py
Paired target/evaluator agents	Patterns	examples/evaluation/001_paired_agents.py
Workflow handoffs and tools	Agent and workflow evaluation	examples/evaluation/002_workflow_evaluation.py
Deterministic, model, and agent mechanisms	Decision table	tests/eval/test_runner.py, tests/eval/test_judges.py
JSONL selection and reproducibility	Datasets	tests/eval/test_dataset.py
CI gate and baseline	Gates and CI	tests/eval/test_cli.py, tests/eval/test_gates.py
PostgreSQL records and jobs	Stores	tests/eval/test_postgres_store.py
RAGAS and plugin extension	Metrics and RAGAS	scripts/smoke_eval_ragas.py, tests/eval/test_ragas.py
Sampled online evaluation	Online evaluation	tests/eval/test_online.py
Privacy, cost, and failures	Privacy and security	tests/eval/test_models.py, tests/eval/test_judges.py
Trace/result correlation	Evaluation telemetry	tests/observability/test_signals.py

The production recipes turn this matrix into complete install, configuration, run, inspection, failure, cleanup, and next-step procedures.

The exact-wheel smoke executes all three credential-free examples. Real PostgreSQL and RAGAS tests run separately because tracing and service lifecycle are part of their contracts.