diff --git a/.gitignore b/.gitignore index aabec74..9da2fa5 100644 --- a/.gitignore +++ b/.gitignore @@ -36,3 +36,4 @@ Thumbs.db # Temporary files *.tmp .cache/ +lib/.precomp/ \ No newline at end of file diff --git a/src/content/docs/adapters/ragas/configuration.md b/src/content/docs/adapters/ragas/configuration.md index fd7c8e3..c8d0f3f 100644 --- a/src/content/docs/adapters/ragas/configuration.md +++ b/src/content/docs/adapters/ragas/configuration.md @@ -50,7 +50,7 @@ Two pre-defined benchmark suites are available: ### `ragas_rag_default`: Default Suite -Runs the four core RAG evaluation metrics. Suitable for most use cases. +Runs the four retrieval-and-grounding metrics. Suitable for most use cases. The fifth core metric, answer correctness (`factual_correctness`), is available via `ragas_rag_full` or by adding it to `parameters.metrics` — see [Metrics reference](metrics/#five-core-ragas-metrics). | Setting | Value | |---------|-------| diff --git a/src/content/docs/adapters/ragas/index.mdx b/src/content/docs/adapters/ragas/index.mdx index 6984462..f87865e 100644 --- a/src/content/docs/adapters/ragas/index.mdx +++ b/src/content/docs/adapters/ragas/index.mdx @@ -10,17 +10,17 @@ The RAGAS adapter integrates [RAGAS](https://github.com/explodinggradients/ragas ## Overview -RAGAS is an open-source framework purpose-built for evaluating RAG pipeline quality. It provides metrics that directly measure whether a RAG system is correctly grounding its answers in retrieved documents: faithfulness, answer relevancy, context precision, context recall, and more. +RAGAS is an open-source framework purpose-built for evaluating RAG pipeline quality. It provides grounding metrics that measure whether a RAG system correctly grounds its answers in retrieved documents—faithfulness, answer relevancy, context precision, and context recall—as well as answer correctness (`factual_correctness`), which scores generated answers against a ground-truth reference. RAG pipelines are among the most widely deployed AI application patterns in enterprise settings, yet faithfulness evaluation, verifying that model-generated answers match the retrieved context rather than hallucinating, remains one of the least commonly performed evaluation steps. The RAGAS adapter brings these metrics into EvalHub's unified evaluation control plane. ### Key Features -- **RAG-specific metrics**: Faithfulness, answer relevancy, context precision/recall, answer correctness, and more +- **RAG-specific metrics**: Faithfulness, answer relevancy, context precision/recall, answer correctness (`factual_correctness`), and more - **LLM-as-judge evaluation**: Uses an LLM judge endpoint for metrics that require semantic understanding - **Embedding-based metrics**: Supports separate embedding model configuration for similarity-based metrics - **Flexible data input**: JSONL and JSON datasets with configurable column mapping -- **Two benchmark suites**: Default (4 core metrics) and Full (11 metrics) evaluation suites +- **Two benchmark suites**: Default (4 retrieval/grounding metrics) and Full (all 11 metrics, including answer correctness) - **OpenAI-compatible**: Works with any OpenAI-compatible model endpoint (vLLM, TGI, Ollama, etc.) ### Supported Backends diff --git a/src/content/docs/adapters/ragas/metrics.md b/src/content/docs/adapters/ragas/metrics.md index c1bcf1c..849b307 100644 --- a/src/content/docs/adapters/ragas/metrics.md +++ b/src/content/docs/adapters/ragas/metrics.md @@ -5,6 +5,38 @@ description: "Reference for all RAGAS evaluation metrics supported by the EvalHu The RAGAS adapter supports 11 evaluation metrics, divided into core metrics (available since RAGAS v0.1) and extended metrics (class-based, available since RAGAS v0.4+). +## Five Core RAGAS Metrics + +RAGAS defines five canonical metrics for RAG pipeline evaluation. The table below maps the common names to the metric IDs used in EvalHub job specs and results: + +| Common name | Metric ID | In `ragas_rag_default` | +|-------------|-----------|------------------------| +| Faithfulness | `faithfulness` | Yes | +| Answer relevance | `answer_relevancy` | Yes | +| Context precision | `context_precision` | Yes | +| Context recall | `context_recall` | Yes | +| Answer correctness | `factual_correctness` | No — use `ragas_rag_full` or add via `parameters.metrics` | + +:::note[Answer correctness vs answer accuracy] +**Answer correctness** is implemented as `factual_correctness`. It checks whether factual claims in the generated answer match the reference answer. Do not confuse it with `nv_accuracy` (Answer Accuracy), which uses a different LLM-based comparison approach. See [Answer Correctness](#answer-correctness-factual_correctness) and [Answer Accuracy](#answer-accuracy-nv_accuracy) below. +::: + +The default benchmark (`ragas_rag_default`) runs the four retrieval-and-grounding metrics. Add `factual_correctness` when you need to score answers against a ground-truth reference: + +```json +{ + "parameters": { + "metrics": [ + "answer_relevancy", + "context_precision", + "faithfulness", + "context_recall", + "factual_correctness" + ] + } +} +``` + ## Metrics Overview | Metric | Category | LLM Judge | Embeddings | Input Requirements | @@ -15,7 +47,7 @@ The RAGAS adapter supports 11 evaluation metrics, divided into core metrics (ava | `context_recall` | Core | Yes | No | `retrieved_contexts`, `reference` | | `answer_similarity` | Core | No | Yes | `response`, `reference` | | `context_entity_recall` | Core | No | No | `retrieved_contexts`, `reference` | -| `factual_correctness` | Extended | Yes | No | `response`, `reference` | +| `factual_correctness` | Core (answer correctness) | Yes | No | `response`, `reference` | | `noise_sensitivity` | Extended | Yes | No | `user_input`, `response`, `retrieved_contexts`, `reference` | | `nv_accuracy` | Extended | Yes | No | `user_input`, `response`, `reference` | | `nv_context_relevance` | Extended | Yes | No | `user_input`, `retrieved_contexts` | @@ -91,14 +123,22 @@ Measures the overlap of named entities between the retrieved context and the ref These metrics use class-based implementations and are available from RAGAS v0.4+. -### Factual Correctness +### Answer Correctness (`factual_correctness`) + +This is the fifth core RAGAS metric. In EvalHub and the RAGAS library it is exposed as `factual_correctness` (class-based, RAGAS v0.4+). + +Measures whether the factual claims in the generated answer match the reference answer. Unlike faithfulness (which checks against the retrieved context), answer correctness checks against the ground truth. -Measures whether the factual claims in the generated answer match the reference answer. Unlike faithfulness (which checks against the context), factual correctness checks against the ground truth. +**When to use**: When you have a reference answer and want to verify that the generated answer is factually correct, regardless of what the context contains. Complements faithfulness: an answer can be faithful to a misleading context but still incorrect against the reference. -**When to use**: When you have a reference answer and want to verify that the generated answer is factually correct, regardless of what the context contains. +**How it works**: The LLM judge extracts factual claims from the generated answer, then verifies each claim against the reference answer. The score reflects the fraction of claims that are supported. **Required columns**: `response`, `reference` +:::tip +Include `factual_correctness` in your metric list or use the `ragas_rag_full` benchmark to evaluate all five core metrics plus extended ones. +::: + ### Noise Sensitivity Measures how sensitive the model is to irrelevant information in the retrieved context. A low noise sensitivity score means the model correctly ignores noisy context; a high score indicates it incorporates irrelevant information into its answers. @@ -109,7 +149,7 @@ Measures how sensitive the model is to irrelevant information in the retrieved c ### Answer Accuracy (`nv_accuracy`) -Measures the accuracy of the generated answer against the reference answer, using LLM-based semantic comparison rather than embedding similarity. +Measures the accuracy of the generated answer against the reference answer, using LLM-based semantic comparison rather than embedding similarity. This is a separate metric from answer correctness (`factual_correctness`): accuracy evaluates overall answer quality, while factual correctness evaluates individual factual claims. **When to use**: When you need a judge-based accuracy score that goes beyond surface-level similarity. Provides a more nuanced assessment than `answer_similarity`.