Evaluator flow and capability policy

For each subject, Praval builds a bounded JudgeContext containing the loaded case, immutable subject observation, and ephemeral target output. The evaluator receives a versioned system rubric plus a canonical candidate envelope. It must return the strict JudgeResult payload: status, score and label for pass/fail, optional bounded explanation, and bounded evidence references.

lease job -> load immutable subject -> resolve ephemeral context
          -> enter evaluation_call_scope
          -> normal Agent tool/model loop with narrowed allowlist
          -> validate structured response
          -> persist idempotent JudgeResult
          -> emit metadata-only linked telemetry

The evaluation-call marker prevents sampled online evaluation from recursively evaluating its own judge calls. It does not hide judge spans; observability can still distinguish and cost them.

Tools and side effects

An allowed tool must already exist on the evaluator agent and satisfy its selected metadata policy. For read_only, use metadata={"read_only": true} or a supported read-only hint. For evaluation_safe, either read-only or metadata={"evaluation_safe": true} is accepted. v0.8.3 rejects evaluator side effects even when a configuration attempts to enable them.

If a tool requires approval, configure evaluator HITL independently. In offline evaluation, the runner may surface the suspended evaluation. In online evaluation, only the durable evaluation job may wait or fail; the completed user request is never reopened.

Isolation and budgets

  • Use a distinct evaluator memory namespace. Never point it at target operational memory.

  • Treat candidate output as untrusted data, not instructions. The rubric and system message must say this explicitly.

  • Cap tool rounds, attempts, wall time, input tokens, output tokens, and cost.

  • Keep temperature at zero for direct judges and pin provider/model profiles.

  • Store evidence as ContentReference values unless an approved content policy requires a protected content store.

Cancellation is re-raised so process shutdown remains correct. Other failures are reduced to a bounded error type; raw provider or tool exception text is not placed in the result.