Benchmark retrieval quality

This is Lesson 5 of the Claude Chat CLI with Hybrid RAG Retrieval series, adapted from the open-source guides in the claude-with-the-anthropic-api repository.

What you’ll build

In this tutorial, you turn six labeled retrieval questions into a repeatable BM25 quality report. You calculate Hit@k, Recall@k, and mean reciprocal rank (MRR), then use the report to identify questions that lexical retrieval misses.

Before you begin

You need:

• Python 3.12 or later.
• uv.
• A local clone with dependencies installed.
• Implement Okapi BM25 from scratch.

You do not need an API key. The benchmark uses local chunks and deterministic labels from eval/rag_cases.jsonl.

See it work

Run the BM25 benchmark from the repository root:

uv run python -m eval.rag_retrieval

Corpus: 267 chunks | Cases: 6 | k=5
Mode      Cases    Hit@k   Recall@k      MRR
bm25          6    0.500      0.417    0.333
bm25 misses: conversation-persistence, citation-deduplication, system-prompt-file

The run searches 267 chunks for 6 labeled questions. The three metrics describe different parts of the resulting rankings.

How retrieval metrics work

Each record in eval/rag_cases.jsonl contains a question and one or more chunk IDs that count as relevant. The benchmark searches for the question, keeps the first k results, and compares those result IDs with the labels.

Hit@k is binary for each case. Recall@k matters when a question has multiple accepted chunks. Reciprocal rank is one divided by the position of the first relevant result; a miss contributes zero. MRR is the mean across all cases.

Load labeled cases

1. In eval/rag_retrieval.py, inspect RetrievalCase:

   @dataclass(frozen=True)
   class RetrievalCase:
       id: str
       query: str
       relevant_chunk_ids: frozenset[str]

   A set of accepted IDs lets one question recognize multiple source chunks as relevant without requiring every chunk to appear.

2. Inspect load_cases() and validate_case_labels(). The loader rejects duplicate case IDs and empty label sets. Label validation rejects chunk IDs that no longer exist after ingestion changes, preventing a stale benchmark from silently reporting false misses.

Score one ranking

1. In score_ranking(), collect the one-based positions of relevant results:

   relevant_ranks = [
       rank
       for rank, chunk_id in enumerate(ranked_chunk_ids, start=1)
       if chunk_id in case.relevant_chunk_ids
   ]

2. Calculate whether the ranking contains a hit:

   hit = bool(relevant_ranks)

3. Divide retrieved relevant chunks by all accepted relevant chunks:

   recall = len(relevant_ranks) / len(case.relevant_chunk_ids)

4. Invert the first relevant rank, or return zero for a miss:

   reciprocal_rank = 1 / relevant_ranks[0] if relevant_ranks else 0.0

The repository implementation also ignores duplicate result IDs before scoring, so one repeated chunk cannot inflate recall.

Aggregate the cases

1. Run every labeled query through the same retrieval mode and cutoff.

2. Average each metric in summarize():

   hit_rate = sum(metric.hit for metric in case_metrics) / case_count
   recall_at_k = sum(metric.recall for metric in case_metrics) / case_count
   mean_reciprocal_rank = (
       sum(metric.reciprocal_rank for metric in case_metrics) / case_count
   )

3. Keep the missed case IDs beside the aggregate values. Aggregate metrics support comparisons, while named misses identify the questions and code paths that need investigation.

Verify your work

Run the focused metric tests:

uv run pytest tests/test_rag_retrieval_eval.py -q

6 passed

Check the benchmark and its tests with Ruff:

uv run ruff check eval/rag_retrieval.py tests/test_rag_retrieval_eval.py

Break it on purpose

Reduce the cutoff from five results to one:

uv run python -m eval.rag_retrieval --top-k 1

Corpus: 267 chunks | Cases: 6 | k=1
bm25          6    0.167      0.167    0.167
bm25 misses: conversation-persistence, citation-deduplication, system-prompt-file, untrusted-rag-context, hybrid-rank-fusion

Hit@1 falls to 0.167 because only one case puts a relevant chunk first. The experiment shows why every retrieval metric must name its cutoff: Hit@1 and Hit@5 describe different user experiences.

Troubleshooting

Next steps

Continue with Test model output with deterministic rules to apply the same repeatable-evaluation mindset to generated responses. Keyed retrieval comparisons belong in a protected environment because they use Voyage embeddings and can incur cost.

For metric definitions and evaluation context, read the Information Retrieval evaluation documentation.