Metrics, plugins, and RAGAS

Built-in deterministic metrics

Metric	Required data	Meaning
terminal_success	subject observation	target ended successfully
exact_match	expected output and target output	canonical JSON equality
tool_call_match	expected tool calls and observed tool facts	exact ordered names
reference.word_overlap	expected output and target output	reference plugin example

All metrics implement the public async Metric.evaluate(JudgeContext) contract and return an immutable MetricResult. Discover installed plugins through the praval.eval.metrics entry-point group. Names and versions must be bounded and unique; plugins cannot shadow built-ins.

RAGAS

Install the optional adapter:

python -m pip install "praval[eval-ragas]"

[models.ragas_judge]
provider = "openai"
model = "gpt-5.4-mini"
temperature = 0.0

[embeddings.ragas_embedding]
provider = "openai"
model = "text-embedding-3-small"

[eval.ragas]
model = "ragas_judge"
embedding = "ragas_embedding"
timeout_seconds = 60
strict_tool_order = true

Supported names are ragas.faithfulness, ragas.response_relevancy, ragas.context_precision, ragas.context_recall, ragas.factual_correctness, ragas.semantic_similarity, ragas.topic_adherence, ragas.agent_goal_accuracy, ragas.tool_call_accuracy, and ragas.tool_call_f1.

Faithfulness and context precision/recall need reference contexts. Response relevancy and semantic similarity need configured embeddings. Tool metrics need expected and observed tool calls. Agent-goal accuracy needs the task input and terminal outcome; its reference variant also needs expected output. Praval validates required fields before a paid call.

RAGAS model and embedding traffic uses configured Praval ModelRuntime and EmbeddingRuntime seams inside the evaluation-call scope. It never chooses a provider, model, credential, or embedding implicitly. Missing fields, timeout, invalid scores, structured-response errors, and provider failures become bounded MetricResult errors. RAGAS, LangChain, and datasets types do not appear in the core praval.eval public API.