Sampling and performance

parentbased_traceidratio is the production default sampler. Make the decision at the workflow root so descendants follow the same decision. Use always_on for small deterministic diagnostics and always_off when an explicitly configured pipeline should emit no spans.

Tune max_queue_size, max_export_batch_size, schedule_delay_millis, export_timeout_millis, and metric_export_interval_millis to the process budget. Smaller queues bound memory more tightly but increase drops during collector outages. Smaller batches reduce latency and usually increase export overhead.

Release certification measures an end-to-end deterministic fake-agent path. No-op instrumentation must remain below 2 percent overhead and enabled, in-memory batched telemetry below 5 percent, excluding network time:

PRAVAL_RUN_PERFORMANCE_TESTS=1 \
  pytest tests/performance/test_observability_overhead.py -v

Do not infer network exporter performance from that test. Load-test the chosen Collector topology and alert on queue depth, dropped items, export failures, and lifecycle failures.

Choosing a ratio

Start with the smallest ratio that still yields enough complete traces for the target’s request volume and error budget. Sampling is all-or-nothing at the root: do not independently sample child agents or Reef consumers. Metrics and structured completion logs remain useful for aggregate monitoring when most traces are unsampled.

always_on is appropriate for deterministic tests and short local diagnosis; it is rarely an acceptable high-volume production default. always_off keeps an explicitly configured pipeline available while suppressing spans, but it does not disable metrics/logs. Disable unneeded signals in observability.otlp when the desired outcome is no exporter at all.

Queue and batch tradeoffs

  • Queue capacity is the hard in-process memory/loss boundary during exporter slowdown.

  • Batch size cannot exceed queue size. Larger batches improve throughput but can increase delivery latency.

  • Schedule delay bounds how long a partially filled trace/log batch waits.

  • Export timeout bounds each exporter attempt; it must fit the service’s total shutdown budget.

  • Metric export interval controls aggregation freshness and backend traffic.

Measure with representative concurrency, tool facts, Reef handoffs, and attribute counts. Benchmark p50/p95/p99 request latency, CPU, resident memory, queue high-water mark, exporter failures, and drops. Never add request IDs, user IDs, raw URLs, prompts, or exception messages as metric dimensions to solve a sampling problem; that creates a cardinality and privacy problem.