# Traces, metrics, and logs

## Traces

Agent and workflow root spans carry stable identity and terminal status.
Producer, delivery, and consumer spans reveal Reef transit. Tool, retry, HITL,
usage, and handoff facts remain bounded on the completed observation.

## Metrics

Praval derives metrics from the same observation: execution invocation,
duration, failure, token usage, tool invocation and duration, retry count, and
Reef handoff count and duration. Dimensions are bounded to low-cardinality
status, kind, agent or workflow name, provider, model, request mode, token type,
tool name, and structured error type.

Pipeline health exposes export attempts, exported items, failures, exceptions,
drops, queue depth and capacity, and lifecycle failures per signal. Inspect
these counters when a backend has gaps; they distinguish application absence
from exporter loss.

| Instrument | Unit | Use |
|---|---|---|
| `praval.execution.invocations` | invocation | completed agent/workflow count |
| `praval.execution.duration` | ms | end-to-end target latency |
| `praval.execution.failures` | failure | non-successful terminal outcomes |
| `praval.gen_ai.token.usage` | token | input/output/total token types |
| `praval.tool.invocations` | invocation | bounded tool status count |
| `praval.tool.duration` | ms | tool latency distribution |
| `praval.retry.count` | retry | retry decisions by operation |
| `praval.reef.handoff.count` | handoff | inter-agent delivery outcomes |
| `praval.reef.handoff.duration` | ms | handoff latency distribution |
| `praval.evaluation.results` | result | judge and metric outcomes |
| `praval.evaluation.score` | 1 | finite normalized score distribution |
| `praval.evaluation.failures` | failure | errored judge/metric results |

Health instruments are `praval.telemetry.export.attempts`,
`praval.telemetry.exported.items`, `praval.telemetry.export.failures`,
`praval.telemetry.export.exceptions`, `praval.telemetry.dropped.items`,
`praval.telemetry.lifecycle.failures`, `praval.telemetry.queue.depth`, and
`praval.telemetry.queue.capacity`, separated by signal.

## Logs

Praval emits a correlated completion log for each observation through the
OpenTelemetry logger provider. Application logs remain application-owned.
Praval neither installs a standard-library logging handler nor captures
arbitrary application messages.

Trace, metric, and log export failures are isolated from the agent request path
and recorded in bounded health state. They do not include payloads or secrets.

## Attribute policy

Stable low-cardinality dimensions include observation kind/status, agent or
workflow name, provider, model, request mode, token type, tool name/status,
operation, handoff status, evaluator name/version, and structured error type.
Run, case, subject, observation, response, trace, and span IDs belong on spans
and log events for correlation; do not group metrics by them in backend
dashboards.

## Backend queries

Backend syntax differs, but the questions should remain portable:

- execution failure ratio: failures divided by invocations, grouped by service
  and agent/workflow;
- p95 duration: the execution-duration histogram by target and status;
- exporter loss: rate of dropped items and export failures by signal;
- tool regression: tool error count and duration by bounded tool name;
- evaluation drift: score distribution and result status by evaluator version.

For Prometheus-compatible backends, prefer `rate()` over counters and
`histogram_quantile()` over the exported histogram buckets. Keep service name
and deployment environment in the OpenTelemetry resource rather than copying
them into every custom application attribute.