The underwriting agent passes every test on the last page. UW-004 is escalate, citing R-ROOF-03. The injected passage of UW-005 changes nothing. UW-007 is decline, because current policy outranks the old approval. Then a pull request arrives: drop the BM25 stage from search_guidance to simplify the pipeline. The author ran the cases. UW-004 still passes: the vector ranking finds R-ROOF-03 without help.
Should it merge? The test suite says yes. It checked three submissions. It did not check the query an underwriter types next week, such as the exact rule id R-SPRK-01. For that query, as this site's case UW-R02 showed, the dense ranking puts the right passage fifth, and with a small context budget it might not reach the prompt at all. A handful of passing cases is evidence about those cases. It says nothing about how often retrieval puts the right passage in front of the writer across the questions people actually ask.
To answer "how often", you need a set of questions with known right answers and a few numbers computed over it. This page builds both: labeled queries, five retrieval metrics computed by hand and then by examples/rag-uw/metrics.py, slices that averages hide, and the difference between measuring retrieval and measuring the answer.
What this is not. This page owns the mechanics of retrieval metrics. Building and maintaining golden sets, graders and rubrics, model judges and the release gates that block a deploy belong to Eval Engineering: see golden datasets, grading and CI/CD quality gates.
Labeled queries: a test suite for retrieval#
Recall the primitive. An embedding column holds one vector per row; a vector query compares the query vector with the rows' vectors, ranks the rows by distance and returns the nearest k; the model reads only what retrieval returned. Retrieval is a query. The model reads only what it returns. So the first thing to measure is whether the query returns the right rows, before any model is involved.
A relevance label is a judgment, made by a person or by a rule written in advance, of how well one row answers one query. The example uses three grades: 2 means the row answers the query, 1 means it is related, 0 means it is not. Unlisted rows count as 0. Here is the first of the example's 14 labeled guidance queries, G01, which is this site's case UW-R01:
G01 "roof replaced recently but aerial imagery shows an older roof"
GD-2026.2-R-ROOF-03 grade 2 (the current rule that answers it)
GD-2025.4-R-ROOF-03 grade 1 (the same rule, superseded version)
GD-NOTE-01 grade 1 (how to read an inspection report)
every other passage grade 0
A set of such queries with their labels is often called qrels (query relevance judgments). The example has 14 guidance queries (data.GUIDANCE_QRELS) and 8 precedent queries (data.PRECEDENT_QRELS). The precedent labels come from row metadata by a rule written before any ranking was seen: for "stated roof age and the provider estimate disagree", a precedent that cites R-ROOF-03 and was not an approval is grade 2, one that cites R-ROOF-03 and was approved is grade 1. Writing the rule first keeps the labels from bending toward whatever the current ranking happens to return.
Before AI. This is a test suite with expected outputs, the same idea as a unit test that asserts a function's return value. The difference is that a search returns a ranked list, not one value, so the assertion becomes a number: how much of the expected set came back, and how high. Search teams built such judgment sets for keyword engines long before embeddings, and the metrics below come from that work.
Five metrics on one ranked list, by hand#
Take one query and one ranking. The example's hand case (metrics.HAND_EXAMPLE) uses the labels of G01 and a ranking of five passages; the cut-off is k = 3.
| Rank | Passage | Label |
|---|---|---|
| 1 | GD-NOTE-01 | 1 |
| 2 | GD-2026.2-R-ROOF-03 | 2 |
| 3 | GD-2026.2-R-ROOF-01 | 0 |
| 4 | GD-2025.4-R-ROOF-03 | 1 |
| 5 | GD-NOTE-02 | 0 |
Three rows are relevant (grade 1 or 2): GD-NOTE-01, GD-2026.2-R-ROOF-03 and GD-2025.4-R-ROOF-03. Two of them are in the top 3.
Recall@k is the share of the relevant rows that appear in the top k. Here 2 of 3 relevant rows are in the top 3: recall@3 = 2 / 3 = 0.667. It answers "did we find what exists?", and it matters most at the candidate stage, because a row not among the candidates can never be reranked or cited.
Precision@k is the share of the top k that is relevant. Here 2 of the 3 rows in the top 3 are relevant: precision@3 = 2 / 3 = 0.667. It answers "how much of what we pass on is useful?", and it matters most at the context budget, where every irrelevant passage takes a slot.
Hit rate@k is 1 if at least one relevant row is in the top k, else 0, averaged over queries. Here it is 1. It answers "did the writer get anything to work with?", and it is forgiving: a list with one weak hit scores the same as a perfect list.
Reciprocal rank is 1 divided by the rank of the first relevant row; MRR (mean reciprocal rank) is its average over queries. Here the first relevant row is at rank 1, so the reciprocal rank is 1 / 1 = 1.0. It answers "how far down must a reader go?", and it ignores everything after the first hit.
nDCG@k (normalized discounted cumulative gain) uses the grades and the positions together. Each row in the top k earns a gain of 2^grade − 1, divided by log2(rank + 1), so higher grades earn more and lower ranks earn less. That sum is DCG. The same sum for the best possible order of the labels is IDCG, and nDCG = DCG / IDCG, a number between 0 and 1.
| Rank | Grade | Gain 2^g − 1 | Discount log2(rank + 1) | Contribution |
|---|---|---|---|---|
| 1 | 1 | 1 | 1 | 1.000 |
| 2 | 2 | 3 | 1.585 | 1.893 |
| 3 | 0 | 0 | 2 | 0.000 |
| DCG@3 | 2.893 |
The ideal order puts the grades 2, 1, 1 first: 3 / 1 + 1 / 1.585 + 1 / 2 = 3 + 0.631 + 0.5 = 4.131. So nDCG@3 = 2.893 / 4.131 = 0.700. Notice what nDCG saw that the others did not. Recall and precision treat the grade-1 note and the grade-2 rule alike, and reciprocal rank stops at the note in rank 1. nDCG lost 30 percent because the row that answers the question sits at rank 2 instead of rank 1, and a relevant row sits below the cut.
Diagram: One ranked list, five questions: what each metric asks of the G01 hand case (k = 3)
The same ranking and labels feed five metrics. Recall and hit rate ask whether the relevant rows were found; precision asks how much of the top k is useful; reciprocal rank asks how high the first hit is; nDCG asks whether the best rows are on top. Values are from the example's hand case.
- Inputs
- Graded labels: A judgment per row and query, made before any ranking is seen.
- Ranked list: What retrieval returned, in order.
- Found and useful
- recall@3 = 0.667: Share of the relevant rows that appear in the top k.
- precision@3 = 0.667: Share of the top k that is relevant.
- hit rate@3 = 1: 1 if any relevant row is in the top k.
- Order and grade
- reciprocal rank = 1.0: 1 divided by the rank of the first relevant row; averaged, MRR.
- nDCG@3 = 0.700: Graded gains discounted by rank, over the ideal order.
The example's code is short enough to read in full. These are the functions from examples/rag-uw/metrics.py, verbatim; rel maps a passage id to its grade.
import math
from typing import Sequence
Labels = dict[str, int]
def _relevant(rel: Labels) -> set[str]:
return {k for k, v in rel.items() if v >= 1}
def recall_at_k(ranking: Sequence[str], rel: Labels, k: int) -> float:
r = _relevant(rel)
return len(r & set(ranking[:k])) / len(r) if r else 0.0
def precision_at_k(ranking: Sequence[str], rel: Labels, k: int) -> float:
return len(_relevant(rel) & set(ranking[:k])) / k if k else 0.0
def hit_rate_at_k(ranking: Sequence[str], rel: Labels, k: int) -> float:
return 1.0 if _relevant(rel) & set(ranking[:k]) else 0.0
def reciprocal_rank(ranking: Sequence[str], rel: Labels) -> float:
for i, rid in enumerate(ranking, start=1):
if rel.get(rid, 0) >= 1:
return 1.0 / i
return 0.0
def dcg_at_k(grades: Sequence[int], k: int) -> float:
return sum((2 ** g - 1) / math.log2(i + 1) for i, g in enumerate(grades[:k], start=1))
def ndcg_at_k(ranking: Sequence[str], rel: Labels, k: int) -> float:
ideal = dcg_at_k(sorted(rel.values(), reverse=True), k)
return dcg_at_k([rel.get(r, 0) for r in ranking], k) / ideal if ideal else 0.0
Run on the hand case (the first line of the script in the slices section below), they return the same numbers as the arithmetic: recall@3 0.667, precision@3 0.667, hit_rate@3 1.0, rr 1.0, dcg@3 2.893, idcg@3 4.131, ndcg@3 0.7. A successful run proves the code and the hand arithmetic agree; it says nothing about whether the labels are right.
The lab: reorder a ranking, watch the arithmetic#
Move passages up or down, change a label (the button cycles 0, 1, 2) or change k. recall@k, precision@k, hit rate@k, reciprocal rank and nDCG@k are recomputed with their arithmetic; DCG and IDCG appear step by step in a collapsible, and a second collapsible holds the averages over the 14 labeled guidance queries. Toy encoders and synthetic data show how the mechanism works, not how good any model or product is.
The recorded case without JavaScript, in this example (ranking as in the table above, k = 3):
| Metric (k = 3) | Arithmetic | Value |
|---|---|---|
| recall@3 | 2 / 3 | 0.667 |
| precision@3 | 2 / 3 | 0.667 |
| hit rate@3 | a relevant row is in the top 3 | 1 |
| reciprocal rank | 1 / 1 | 1.000 |
| nDCG@3 | DCG 1 + 3 / log2(3) = 2.893; IDCG 3 + 1 / log2(3) + 1 / 2 = 4.131 | 0.700 |
Try one move: swap ranks 1 and 2. Recall, precision, hit rate and reciprocal rank do not change. nDCG rises from 0.700 to 0.879 (DCG 3 + 1 / 1.585 = 3.631, over the same IDCG 4.131), because the grade-2 row is now first. It does not reach 1.0, because the third relevant row still sits below the cut. That is the reason to report nDCG when grades exist.
From one query to a labeled set#
One query is an anecdote. The example's run_demo.py computes the five metrics over all 14 guidance queries for three retrieval modes, and over the 8 precedent queries, at k = 5. This is the metrics part of its output from a scratch copy, unchanged; the 13 case lines and 8 debugger lines before and after it are left out.
cd rag-uw && /opt/homebrew/bin/python3 run_demo.py
...
Retrieval metrics on the labeled queries (k = 5; synthetic labels, toy encoder):
guidance lexical recall@5=0.780 precision@5=0.357 hit_rate@5=1.000 mrr=0.964 ndcg@5=0.801
guidance dense recall@5=0.774 precision@5=0.357 hit_rate@5=0.857 mrr=0.798 ndcg@5=0.691
guidance hybrid recall@5=0.893 precision@5=0.414 hit_rate@5=1.000 mrr=0.917 ndcg@5=0.793
precedents dense recall@5=0.447 precision@5=0.650 hit_rate@5=0.875 mrr=0.896 ndcg@5=0.725
...
These numbers are from this example: synthetic labels, a toy encoder and an 18-passage corpus. They describe the example's mechanics, not any model. With that said, read them as an engineer would read a test report.
Back to the pull request. Dense alone has a hit rate of 0.857: for 2 of the 14 queries, nothing relevant reached the top 5. Hybrid has 1.000 and the highest recall@5 (0.893). The proposal to drop BM25 would turn two queries from "something to cite" into "nothing to cite", while UW-004 kept passing. Lexical alone has the highest MRR and nDCG, because many of these queries share words with their passages; hybrid trades a little of that for recall. Which mode wins depends on which number the next stage needs, which is why the pipelines page measures recall at the candidate stage and precision at the context budget.
Precision@5 looks low everywhere (0.357 to 0.414), and that is not a flaw. Most guidance queries have only two or three relevant passages, so even a perfect ranking cannot fill five slots with relevant rows. Precision@k has a ceiling of (relevant rows) / k when fewer than k rows are relevant. The precedent line shows the opposite case: many precedents are relevant to each query under the metadata rule, so precision@5 is high (0.650) while recall@5 is low (0.447), because five slots cannot hold all of them.
Averages also hide the queries that matter. The script in the next section prints three queries one by one. For G02, the exact id R-SPRK-01, dense retrieval has a reciprocal rank of 0.167 (the first relevant passage at rank 6) and an nDCG@5 of 0.000, while lexical and hybrid both have a reciprocal rank of 1.0. One bad query in fourteen moves an average a little; in production it is every underwriter who types a rule id.
Slices: where an average lies#
A slice is a subset of the labeled queries, or of the rows, chosen because it can fail differently from the rest. The two that matter most for this agent are corpus slices (profiles versus precedents, guidance versus precedents) and version slices (current versus superseded rows).
The script below, a short file in a scratch copy of examples/rag-uw, prints the hand case, three queries one by one, and a version slice: across the 14 guidance queries, how many superseded passages reach the top 5 without a version filter and with the filter policy_version = '2026.2', and recall@5 against labels restricted to current passages. It calls only the example's metrics, data and the ranking helper run_demo._rankings. A successful run proves what each mode returns on these labels; it does not prove anything about a model.
# eval_slices.py, run from a scratch copy of examples/rag-uw
# (1) the hand example through metrics.py; (2) per-query numbers that an average hides;
# (3) a current-vs-superseded slice: how many superseded passages reach the top 5, per mode.
import data
from metrics import evaluate, hand_example_values, ndcg_at_k, reciprocal_rank
from run_demo import _rankings
print("1. hand example (k = 3):", {k: round(v, 3) for k, v in hand_example_values().items()})
ALL = {"trusted": True} # no version filter, as in run_demo's metrics
CUR = {"trusted": True, "policy_version": data.CURRENT_VERSION}
status = {p["passage_id"]: p["status"] for p in data.guidance_passages()}
qrels = {q["qid"]: q["rel"] for q in data.GUIDANCE_QRELS}
rk_all = {q["qid"]: _rankings(q["query"], ALL) for q in data.GUIDANCE_QRELS}
rk_cur = {q["qid"]: _rankings(q["query"], CUR) for q in data.GUIDANCE_QRELS}
print("2. per query, no version filter (RR = reciprocal rank):")
for qid in ("G01", "G02", "G03"):
row = [f"{m} RR={reciprocal_rank(rk_all[qid][m], qrels[qid]):.3f} nDCG@5={ndcg_at_k(rk_all[qid][m], qrels[qid], 5):.3f}"
for m in ("dense", "lexical", "hybrid")]
print(f" {qid}: " + " | ".join(row))
print("3. slice: superseded passages in the top 5, summed over 14 queries")
for mode in ("dense", "lexical", "hybrid"):
stale_all = sum(status[p] == "superseded" for q in rk_all for p in rk_all[q][mode][:5])
stale_cur = sum(status[p] == "superseded" for q in rk_cur for p in rk_cur[q][mode][:5])
cur_labels = {q: {p: g for p, g in rel.items() if status[p] == "current"} for q, rel in qrels.items()}
r_all = evaluate({q: rk_all[q][mode] for q in qrels}, cur_labels, 5)["recall@5"]
r_cur = evaluate({q: rk_cur[q][mode] for q in qrels}, cur_labels, 5)["recall@5"]
print(f" {mode:8} no filter: {stale_all:2} superseded, recall@5 on current labels {r_all:.3f}"
f" | filter {data.CURRENT_VERSION}: {stale_cur} superseded, recall@5 {r_cur:.3f}")
1. hand example (k = 3): {'recall@3': 0.667, 'precision@3': 0.667, 'hit_rate@3': 1.0, 'rr': 1.0, 'dcg@3': 2.893, 'idcg@3': 4.131, 'ndcg@3': 0.7}
2. per query, no version filter (RR = reciprocal rank):
G01: dense RR=1.000 nDCG@5=0.758 | lexical RR=1.000 nDCG@5=0.242 | hybrid RR=1.000 nDCG@5=0.363
G02: dense RR=0.167 nDCG@5=0.000 | lexical RR=1.000 nDCG@5=0.700 | hybrid RR=1.000 nDCG@5=0.605
G03: dense RR=1.000 nDCG@5=0.689 | lexical RR=0.500 nDCG@5=0.659 | hybrid RR=0.500 nDCG@5=0.587
3. slice: superseded passages in the top 5, summed over 14 queries
dense no filter: 21 superseded, recall@5 on current labels 0.774 | filter 2026.2: 0 superseded, recall@5 0.929
lexical no filter: 21 superseded, recall@5 on current labels 0.774 | filter 2026.2: 0 superseded, recall@5 0.881
hybrid no filter: 28 superseded, recall@5 on current labels 0.869 | filter 2026.2: 0 superseded, recall@5 0.929
The per-query lines show that no mode wins everywhere. Dense scores highest on G01, the paraphrase of UW-R01, and lowest on G02, the exact id. Lexical is the reverse. That is the argument for hybrid search, made with numbers.
The version slice holds a lesson about labels, not about retrieval. The run_demo.py metrics run without a version filter, and the labels give superseded passages grade 1 ("related"). So in the averages above, a superseded rule in the top 5 counts as a hit. The slice shows the cost: without the filter, 21 to 28 of the 70 top-5 slots go to superseded passages, and recall against current passages is lower, because stale rows take the slots. With the filter, no superseded passage appears and recall on current labels rises to 0.929 for dense and hybrid. A metric rewards what its labels reward. If a superseded rule must never be cited, either grade it 0 for the decision use case or measure the current slice separately, as here. Case UW-R03 is this slice in one query.
The corpus slices need their own labels. The precedent line of run_demo.py is one corpus slice, measured on the active book-east rows the label rule is defined on. Corpus A has no relevance labels in the example. Its quality question is different (is a neighbor a true comparable?), and the two corpora page measures it as a false-match count against a rule written for that page: 3 of 5 for the cosine ranking of SUB-1004's profile, 1 of 5 for a weighted band distance. Report profile and precedent numbers side by side, never pooled, because one corpus can improve while the other gets worse.
Index recall is not recall@k#
Two numbers share the word recall, and confusing them leads to wrong conclusions. Index recall compares an approximate index with exact search: of the exact top k, the share the index also returned. Recall@k compares a ranking with human labels: of the rows a person marked relevant, the share in the top k. Exact search is the ground truth for the first; labels are the ground truth for the second.
They fail independently. In this site's case UW-R10, a toy HNSW over 4,000 precedent vectors reached index recall@10 of 0.67 at ef_search 10 and 1.00 at 100; on the throwaway PostgreSQL database, Q11 measured index recall@10 of 1.0 at ef_search 10, 40 and 100 on a 20,000-row toy table (one run on one laptop, toy data; not a benchmark). An index recall of 1.0 means the index returned exactly what exact search would have. If the embedding puts the wrong rows nearest, as dense retrieval does for G02, the index faithfully returns the wrong rows, and recall@k against labels stays low. The ANN page measures index recall; this page measures whether the rows were the right ones. Tune an index with the first, choose an embedding or a fusion with the second.
Latency is a measurement, not a guess#
A retrieval change can improve every metric above and still be wrong for production because it is too slow. Latency belongs in the same report, measured, never estimated. This page reports no latency numbers, because the example is toy Python and its timings would mean nothing. What to measure is still clear: the duration of each stage, recorded in the retrieval trace next to its row counts; the 95th percentile per stage under the expected query rate, not the average of the whole call; and a count of work that does not depend on the machine, such as the example's distance computations. The scale page runs that playbook as the corpus grows.
Beyond retrieval: faithfulness and answer correctness#
Retrieval metrics stop at the ranked list. The writer still has to use it. Two more properties are measured on the answer.
Faithfulness is whether every claim in the answer is supported by the passages and evidence it was given. Answer correctness is whether the answer is the right one for the case. They are different, and the debugger shows it. In the wrong_citation scenario, the broken writer recommends escalate, which is correct for UW-004, but cites R-ROOF-01, which does not support that recommendation for a 22-year roof: correct but unfaithful. The opposite also happens: a writer that reads a superseded passage and faithfully applies its 30-year limit is faithful to its context and wrong for the case, and the fault is retrieval's (case UW-R03).
Diagram: Retrieval-only and end-to-end evaluation measure different things
Retrieval-only evaluation scores the ranked list against labels and explains failures. End-to-end evaluation scores the agent's answer: is it faithful to its passages, and is it correct for the case. Human review checks both and feeds new labels back.
- Retrieval-only
- Labeled queries: Questions with graded labels written before any ranking.
- Metrics per slice: Scores per corpus and per version, not only averages.
- End-to-end
- Faithfulness: Every claim is supported by the passages and evidence given.
- Answer correctness: The recommendation, citations and fields match the expected answer.
- Human review
- Underwriter decisions: Every decision on a recommendation is a label; overrides teach most.
So there are two kinds of evaluation, and a team needs both. Retrieval-only evaluation runs the query and scores the ranked list against labels: fast, cheap, deterministic, and blind to what the writer does. End-to-end evaluation runs the whole agent on a case and scores the output: UW-004 must be escalate, cite R-ROOF-03, list EV-ROOF-0412, have requires_human true and policy_version 2026.2. End-to-end catches writer failures; retrieval-only explains them. When an end-to-end case fails, the retrieval metrics for its query say whether the right passage was ever in the context. When retrieval recall drops and end-to-end cases still pass, the cases are probably too few.
For deterministic parts, the checks are code: the recommendation equals the expected value, every citation is in the context, requires_human is true. For free-text reasons, faithfulness needs a grader, a person or a model judge, and a model judge has its own error rate that must be measured; that discipline is covered at eval.sammuthu.com.
In the insurance build. In that build, retrieval is evaluated with golden sets in JSON, scored by hit@1, hit@3, MRR and filter precision. The runs are stored, and a test fails the build when a metric regresses.
Human review closes the loop#
Labels and graders are proxies. The underwriter is the ground truth the agent exists to serve. Every decision an underwriter makes on a recommendation is a label: accepted as written, accepted with a different reason, or overridden. Overrides, like PRE-002's decline after an inspection found storm damage, are the most informative, because they mark cases where the agent's evidence or reasoning fell short.
Two practices turn that into evaluation. Sample decided cases for review, and add the ones that reveal a retrieval gap to the labeled set, with labels written by a person who did not see the ranking. And track override rates per slice (per rule, per corpus, per policy version) over time, so a regression shows up as a number before it shows up as a complaint. How to run a review program and turn its results into golden-set entries is on grading.
The lab: eight failures and the test that catches each#
Each failure in the debugger enters at one stage and has one test that fails on the broken run and passes on the fixed one. Read each test as an evaluation case: it is the smallest check that would have caught the failure before release. Toy encoders and synthetic data show how the mechanism works, not how good any model or product is.
The same eight scenarios without JavaScript, in this example:
| Failure | Case | Step at fault | Test (fails broken, passes fixed) |
|---|---|---|---|
| Stale policy in the context | UW-R03 | filter | No superseded passage in the selected context |
| Inadequate recall: k too small | UW-004 | candidates | The recommendation is escalate and cites R-ROOF-03 |
| A chunk holds half a rule | UW-R08 | chunk | The top chunk contains the R-ROOF-03 heading and its last sentence |
| Another tenant's precedent leaks | UW-R07 | filter | Every returned precedent belongs to the querying tenant |
| An instruction inside a retrieved passage | UW-005 | generate | Decline citing R-ROOF-01, requires_human true |
| Unrelated passages fill the context | none | select | Every context passage meets the threshold (empty context: abstain) |
| The citation names the wrong rule | UW-004 | generate | Citations equal R-ROOF-03 and the cited passage is in the context |
| Query and index from different encoders | UW-R06 | encode | The dense top 5 equals the exact top 5 with the index's own encoder |
Four of the eight are retrieval failures a metric would show (stale policy, low recall, bad chunk, mismatch); two are permission or relevance failures a slice would show (leak, low relevance); two are writer failures only an end-to-end check catches (injection, wrong citation).
In a coding agent. A coding agent's retrieval can be labeled the same way. Take twenty questions developers asked ("where do we retry provider calls", "what calls search_code"), mark the files and functions that answer each, and compute recall@10 for grep, for the vector index and for both fused. Slice by question type: identifier questions favor grep, behavior questions favor vectors. Re-run after a large refactor, because labels point at paths that move.
A checklist for Monday#
- Write 20 to 50 labeled queries per corpus with graded labels, including exact identifiers, paraphrases and questions the corpus cannot answer.
- Write label rules before looking at any ranking; record who labeled and when.
- Compute recall@k at the candidate stage, precision@k and nDCG at the context budget, hit rate and MRR for the first useful row.
- Report per query and per slice (corpus, current versus superseded, tenant), not only averages.
- Keep index recall (against exact search) and recall@k (against labels) in separate columns.
- Record per-stage latency in the trace and look at the 95th percentile; never quote toy timings.
- Run end-to-end cases for the recommendation, citations, evidence ids and
requires_human, next to retrieval-only metrics. - Feed underwriter overrides back into the labeled set.
Where labels come from, and how they go wrongPooling, label rules, graded versus binary, and the drift of a labeled set
Labeling every row for every query is impossible beyond toy corpora. The usual approach pools: run several retrieval methods, take the union of their top results, and label only those. Pooling has a known bias: a relevant row that no method retrieved is never labeled and counts as 0, so a new method that finds it is not rewarded. Mitigations are to pool generously, to include a method unlike the others (lexical next to dense), and to add rows found later through review.
Label rules from metadata, as the example uses for precedents, are cheap and consistent, and they encode the rule's author's view of relevance. They are best for properties the metadata really captures (rule cited, decision, override) and weakest for nuance. Graded labels carry more information than binary ones, and nDCG uses it, but graders disagree more on grades; measure agreement on a sample before trusting the second decimal. Finally, labels age: when the policy moves to a new version, grade-2 passages become superseded, and a labeled set that is not updated starts rewarding stale answers, the version slice above in slow motion.
How much a small labeled set can tell youFourteen queries are a smoke test, not a significance test
The example's guidance set has 14 queries. One query moving from a hit to a miss changes hit rate by 1/14, about 0.07. A difference of that size between two modes is an observation about those 14 queries, not evidence that one mode is better in general. With small sets, look at per-query changes first: which queries got better, which got worse, and why. A rule set in advance, such as "any query that was a hit and became a miss must be explained before merging", is an investigation trigger, not a statistical claim.
Larger sets and repeated measurement are what support comparisons. For deterministic retrieval, repeated runs give the same numbers, so variation comes from the choice of queries; resampling the queries (a bootstrap) gives a sense of how much the average would move with a different sample. For stages with a model in them, run each case several times and report the spread. How confidence and uncertainty are reported across trials is covered at eval.sammuthu.com.
nDCG has variants; say which one you useGain, discount and the ideal list change the number
The measure was proposed by Järvelin and Kekäläinen (ACM TOIS, 2002). Their paper defines cumulated gain as the sum of graded relevance scores down the list, discounted cumulated gain as dividing a document's score by the logarithm of its rank in base b, with no discount for ranks below b, and a normalized form that divides by the ideal vector. The example uses a later common form: gain 2^grade − 1 and discount log2(rank + 1), so the top rank is not discounted and grade 2 is worth three times grade 1. With linear gain (gain = grade), the hand case gives a different value.
Two more choices change the number. The ideal list can be built from all labeled rows or only from retrieved ones; the example uses all labels, which is stricter. And queries with no relevant rows have no defined nDCG; the example returns 0 for them, other tools skip them. None of these is wrong, but numbers computed with different choices are not comparable. Write the formula next to the metric in every report.
Evaluating abstention and unanswerable questionsA labeled set with only answerable queries cannot measure the threshold
Every query in the example's guidance set has at least one relevant passage. That makes recall and nDCG well defined, and it hides a whole class of behavior: what happens when nothing relevant exists. The marina query on the grounding page is such a case; the right output is an abstention.
Add unanswerable queries to the set and score them separately: the share of unanswerable queries on which the writer abstained, and the share of answerable queries on which it wrongly abstained. Those two numbers are what an abstention threshold trades, and they move in opposite directions. Report them with the retrieval metrics, per slice, and recompute after any change to the encoder, the reranker or the threshold.
In one breath#
A retrieval change is judged on labeled queries, not on a few passing cases: graded labels written before any ranking, then recall@k (did we find what exists), precision@k (how much of what we pass on is useful), hit rate (did we find anything), MRR (how high is the first hit) and nDCG (are the most relevant rows on top), each computed by hand once and by metrics.py after. Averages hide slices: dense retrieval fails the exact-id query G02, and unfiltered rankings fill three-tenths to two-fifths of the top-5 slots with superseded passages that the labels quietly reward. Index recall checks an index against exact search; recall@k checks the rows against people; faithfulness and answer correctness check the writer, and underwriters' overrides check everything.
The insurance agent now has a way to say how often it finds the right evidence and whether its answers rest on it. What is still open is where all of this runs in production: which managed database, index service or retrieval product holds the three corpora, and what each one changes about filters, updates and evaluation. The next page, Google Cloud, compares those products from their documentation, dated and without ranking.