The underwriting agent's retrieval works for the default case. For UW-004 the pipeline encodes the query, filters on the current policy version, retrieves 50 candidates with a keyword leg and a vector leg, fuses them, reranks to 10 and puts 5 passages in the context; R-ROOF-03 is among them, and the agent recommends escalate. The model recommends; a person decides.
Then the team collects the queries that went wrong over a month, and three kinds stand out. A broker writes "the roof on this one looks tired, paperwork says 9", and nothing in those words resembles the policy's language. An underwriter asks whether a 2025 approval was right at the time, and retrieval returns the current 25-year rule instead of the 30-year rule in force then: a variant of the series case UW-007. A question about who resolves a roof-age conflict retrieves half of R-ROOF-03, because the passage that answers it is long.
Each of these has a named technique in the literature and in products, and each technique costs something: more model calls, more storage, more moving parts or less predictability. The standard on this page is a rule: add a technique only when a labeled slice of queries shows the failure it addresses, and keep it only if an evaluation against the baseline shows the fix. Everything below is measured against the pipeline you already have.
What this is not. Choosing and ordering what enters the context window is context.sammuthu.com; retrieval exposed to the model as a tool, with its schema, is tools.sammuthu.com. This page covers what changes inside retrieval itself.
The baseline, and where each technique plugs in#
Retrieval is a query. The model reads only what it returns. In plain words: an embedding column holds one vector per row, a query vector is compared with them, the rows are ranked by distance, and the nearest k are returned; the pipelines page lays out the stages. Every advanced technique changes one of three things: the query that is sent, the units that are stored and searched, or the conditions a row must meet.
Diagram: Where each advanced technique changes retrieval: the query, the stored units, or the filter
Read the three lanes as three places to intervene. Query-side techniques transform what is searched and spend model calls on every query. Index-side techniques change what is stored, so they need re-ingestion and often re-embedding. Temporal retrieval adds a date condition inside the query. All of them feed the same ranking and the same context budget.
- The query: cost on every query
- Rewrite or expand: A model call turns the asked text into the searched text.
- Multi-query and HyDE: Several searches fused by rank, or one search with the vector of a hypothetical answer.
- Agentic retrieval: The model chooses when and what to search, and may search again.
- The stored units: cost at ingestion
- Parent-child and hierarchy: Small units are searched; their parent section, or a level below a coarse match, is read.
- Late interaction: ColBERT-style MaxSim over token vectors; storage grows with tokens.
- Multimodal and graph: New kinds of stored units: image vectors, or entities and their relations.
- The filter: a date inside the query
- Temporal retrieval: Effective dates on every version; the query says which date it means.
The diagram sorts the ten techniques by that rule. Query-side techniques (rewriting, multi-query, HyDE, agentic retrieval) leave the corpus alone and spend model calls at query time. Index-side techniques (parent-child, hierarchical, late interaction, multimodal, graph) change what is stored, so they need re-ingestion and often re-embedding. Temporal retrieval changes the filter. That split predicts the cost: query-side techniques add latency to every query; index-side techniques add work to every ingestion and migration.
Before AI. Search engines met the first problem decades ago. A Lucene index with a synonym filter expanded "roofing" to "roof" at query time; a thesaurus mapped "auto" to "car"; "did you mean" rewrote misspellings. Query rewriting with a model is the same idea with a larger vocabulary, and the same risk: an expansion that drifts from what the person meant.
Query rewriting and expansion#
The problem. The person's words and the corpus's words differ. "Roof looks tired, paperwork says 9" shares almost nothing with "Roof age on the application and from the provider differ by more than 5 years".
The mechanism. Before retrieval, a model rewrites the query into the corpus's register, or adds terms to it. Query rewriting is that step: the text that is searched is a transformed version of the text that was asked. For the insurer, the broker's phrase becomes "roof age on the application differs from the provider's assessment".
The cost. One model call per query, before retrieval can start, so it adds to latency. The rewrite can also drop a constraint, such as a wind zone, or add one that was never asked.
When not to use it. When the query already comes from a template or a program, as the agent's tool calls usually do; when the embedding already folds the paraphrases (the toy encoder folds "aged roofing" and "older roof" into one vector); or when identifiers must pass through untouched. Keep the original query as one of the searches, and log the rewrite in the trace.
Multi-query#
The problem. One phrasing of a question finds one neighborhood of the space; a relevant passage written differently may sit right outside the top k.
The mechanism. Generate several phrasings, run one retrieval for each, and fuse the ranked lists by rank with reciprocal rank fusion, as on hybrid. Multi-query is that fan-out: a passage that ranks well under several phrasings rises.
The cost. N searches instead of one, plus the model call that writes the phrasings. The run below shows the cost without a gain in this toy: three phrasings of the UW-R01 query, each ranking R-ROOF-03 first or second, fuse to the same top two as the single query.
When not to use it. When the single query already ranks the relevant rows inside the candidate pool. Multi-query raises recall at the candidate stage; if recall@50 is already high, it buys latency and nothing else.
HyDE: search with a hypothetical answer#
The problem. A short question and a long, formal passage sit far apart in the space even when one answers the other.
The mechanism. HyDE, from the paper "Precise Zero-Shot Dense Retrieval without Relevance Labels" (Gao, Ma, Lin and Callan, submitted 20 December 2022), has a language model write a hypothetical document that answers the question, encodes that document, and retrieves the real documents nearest to its vector. For the insurer, the model drafts a passage such as "If the roof age stated by the broker and the age from the provider differ by several years, a senior underwriter should review". Its vector lands near R-ROOF-03.
The cost. A model call that writes a whole passage on every query. And a risk that needs a rule: the hypothetical passage is invented text, possibly with an invented threshold or rule id. It must be used only as a query vector, never placed in the context or cited. Only retrieved passages are evidence.
When not to use it. When queries and documents are already in the same register, when latency is tight, or when the hypothetical text could be confused with evidence anywhere downstream.
Parent-child retrieval#
The problem. Small chunks are sharp to search but poor to read; large chunks are the reverse. This site's case UW-R08 shows the cost of the wrong cut: a chunk ends in the middle of R-ROOF-03, and the half that names who resolves the conflict is missing.
The mechanism. Search over small child chunks, then hand the model the larger parent each one belongs to, such as the rule section. Parent-child retrieval decouples the unit that is searched from the unit that is read. The chunking page introduced the idea; the run below shows it: three small children match, two of them inside R-ROOF-03, and the model receives two whole sections, with the rule complete.
The cost. Bookkeeping: each child stores its parent id, parents are versioned with the same care, duplicates must collapse (two children, one parent), and the context budget is spent on larger units.
When not to use it. When the corpus already has natural units that fit the budget. A policy with explicit sections can use the section-aware splitter and retrieve sections directly; Corpus A and Corpus B records are already one unit each.
Hierarchical retrieval#
The problem. In a large corpus with structure, searching every leaf for every query is wasteful and noisy: the right section is easy to find, the right sentence inside it is not.
The mechanism. Search in levels: first a coarse level (documents, sections, or summaries of them), then only the leaves under the winners. The levels can be the document's own structure or written summaries. A plain structured filter is the simplest hierarchy: in the insurance build, precedent retrieval first restricts to one provision, status = 'active' and a 36-month lookback, then sorts that small set by distance. That is hierarchy expressed as a filter, with no second index.
The cost. An error at the top level is unrecoverable: if the coarse step picks the wrong section, no leaf search can bring the right one back. Summaries must be regenerated when the text changes.
When not to use it. When the corpus is small enough to search exactly, as the example's 18 guidance passages are, or when a metadata filter already expresses the hierarchy.
Late interaction#
The problem. One vector per passage compresses every word into a single point. A query that hinges on one exact token, such as R-SPRK-01 in UW-R02, gets blurred with the rest.
The mechanism. Keep one vector per token instead of one per passage. The ColBERT paper (Khattab and Zaharia, SIGIR 2020) encodes query and document separately; then every query token vector is compared with all document token vectors by a MaxSim operator (the maximum similarity, for example cosine), and those maxima are summed across the query's tokens. Document encodings can be computed offline, and the paper uses the approach both to re-rank candidates and to retrieve directly from a large collection with a vector index. It is called late because query and document meet only after both are encoded, unlike a cross-encoder, which reads them together, as on reranking.
The cost. Storage multiplies by the number of tokens per passage: a passage of 100 tokens stores 100 vectors instead of one. Products differ in where they use it: Vespa documents ColBERT tensors scored with MaxSim for ranking, not for the approximate retrieval step, and Weaviate added multi-vector support, naming ColBERT, in v1.29 (ecosystems has the details, as read on 2026-11-18).
When not to use it. When hybrid search with BM25 already catches the identifiers, which is the cheaper fix for UW-R02, or when storage and memory are tight.
Multimodal embeddings#
The problem. Some evidence is not text: roof photos, inspection images, scanned forms.
The mechanism. A multimodal embedding model maps more than one kind of input, for example images and text, into one space, so a text query can retrieve an image and the reverse. For the insurer, a query "roof covering in poor condition" could retrieve inspection photos next to the guidance passages. This site's source pack holds no primary source for a specific multimodal model, so this section describes the general idea only.
The cost. A different model with its own migration path; larger inputs to embed; and evidence that a person must still be able to inspect, with an evidence id like EV-ROOF-0412.
When not to use it. When text extracted from the image (an inspection report's findings, OCR of a form) carries the facts the rules need. A rule that compares roof ages needs the number, and numbers are what embeddings see worst.
Graph RAG#
The problem. Some questions follow relations across many records: which precedents cite a rule that changed, and which of those were overrides.
The mechanism. Store entities and their relations as a graph (submission, rule, precedent, policy version), retrieve by walking the relations, and give the model the connected facts. The term covers several published designs; this site's source pack holds none of their papers, so the description stays general.
The cost. Building and maintaining the graph: extracting entities and relations, often with a model, and keeping them current as records change.
When not to use it. When the relations already live in a schema. In the example, "precedents joined to the current text of the rules they cite" is a SQL join: the example's Q5 returns PRE-001, PRE-002 and PRE-004 with the current rule passage for each, from foreign keys and ANY (pr.rule_ids). A graph earns its cost when the relations are buried in unstructured text across documents, not when they are columns.
Temporal retrieval#
The problem. The right answer depends on a date. The current policy 2026.2 is effective from 2026-07-01; the earlier 2025.4, from 2025-01-01, set the roof limit at 30 years. A question about today needs the current rule; a review of a 2025 decision needs the rule in force then.
The mechanism. Store effective dates on every version and filter "as of" a date: the latest version whose effective date is on or before it. Temporal retrieval is retrieval with that date condition inside the query, next to the vector ranking. The run below asks which R-ROOF-01 passage was in force on 2025-06-06, the decision date of precedent PRE-023, and on the example's "today": 2025.4 and 2026.2. That is how UW-007 stays correct: the 2025 approval is judged against the rule of its time for the review, and current policy outranks it for any new recommendation.
The cost. Effective dates must be recorded and trusted at ingestion, and every query must say which date it means. The default must be "now", and the date must come from the application, never be guessed by the model.
When not to use it. When there is only one version, or when superseded text should never be retrievable at all; then filter status = 'current', as UW-R03 does.
Agentic retrieval#
The problem. Some questions cannot be answered with one retrieval: the first results reveal what to look for next.
The mechanism. The model decides when to retrieve and with what query, by calling retrieval as a tool, reading the results, and calling again. Agentic retrieval is that loop. The underwriting agent already works this way: it calls search_guidance(query) and find_similar_decisions(submission_id), and may search again after reading a precedent decided under an old version.
Diagram: Agentic retrieval: the model chooses the query, the application owns the filters
Time runs down the page. The model asks for a retrieval with query text only. The application adds the tenant and policy-version filters from the session, enforces the budgets and records the call in the trace. The model reads the results and may search again. The recommendation goes to a person, who decides.
- Underwriting agent (model) to Application (harness): search_guidance(query). The model supplies query text only; it has no parameter for tenant or version.
- Application (harness) to Retrieval (corpora): query + session filters. The application binds tenant and policy version and enforces the budgets.
- Retrieval (corpora) to Application (harness): passages with ids. Ranked rows; the call is written to the retrieval trace.
- Application (harness) to Underwriting agent (model): results. The model reads only what retrieval returned.
- Underwriting agent (model) to Application (harness): find_similar_decisions(submission). A second retrieval chosen after reading the first.
- Underwriting agent (model) to Underwriter: recommendation with citations. The model recommends; a person decides.
The diagram marks the boundary that keeps the loop safe. The model chooses the query text; the application adds the tenant and policy-version filters from the session and enforces the budgets. The model never sees a parameter that would let it widen its own access. Every call lands in the retrieval trace.
The cost. More model calls and retrievals per question, latency that varies from question to question, and results that are less predictable to evaluate: the same question can take two paths.
When not to use it. When one well-built retrieval answers the question, which is most of the time. Tool design (schemas, descriptions, what the model is allowed to pass) is the subject of tools.sammuthu.com.
In a coding agent. A coding agent is agentic retrieval in its plainest form: it greps for an identifier, reads a file, then searches for the callers it found. Multi-query helps "where do we retry provider calls" find code written as "backoff" or "attempts"; parent-child returns the whole function around a matching line; grep stays the right tool for an exact symbol, where late interaction or a rewrite would only add cost.
Three techniques, run on the example#
The program below runs multi-query with fusion, parent-child retrieval and an as-of filter on the example's own data. It demonstrates each mechanism and its bookkeeping. A successful run proves the mechanism behaves as described on these 18 guidance passages with a toy encoder; it does not measure any gain a real model would see. The three rewrites are written by hand, standing in for a model.
# advanced_demo.py, run from a scratch copy of examples/rag-uw
# Three advanced techniques on the example's own data: multi-query with RRF, parent-child retrieval,
# and as-of (temporal) retrieval. Toy encoder, synthetic data; the rewrites are written by hand here,
# where a production system would ask a model for them.
from chunker import recursive_split, section_split
from data import AS_OF, guidance_document, guidance_passages
from encoders import TOY_A1
from fusion import rrf
from search import VectorTable, search_text
current = [p for p in guidance_passages() if p["status"] == "current"]
table = VectorTable.build("guidance", current, "passage_id", "text", TOY_A1)
# 1. Multi-query: several phrasings, one ranked list each, fused by rank.
rewrites = ["roof replaced recently but aerial imagery shows an older roof",
"the broker says the roof is new but the provider says it is old",
"two sources give different roof ages"]
lists = {f"q{i}": [h["id"] for h in search_text(table, q, 5).hits] for i, q in enumerate(rewrites)}
fused = [r["id"] for r in rrf(lists)][:5]
for name, ids in lists.items():
print(f"1. {name} top 2: {ids[:2]} rank of R-ROOF-03: {ids.index('GD-2026.2-R-ROOF-03') + 1}")
print(f" fused top 2: {fused[:2]} searches run: {len(rewrites)}")
# 2. Parent-child: search small chunks, hand the model the section each one belongs to.
doc = guidance_document()
children = recursive_split(doc, 150, 30)
parents = section_split(doc, 1000, 0)
rows = [{"id": f"C{c.ordinal:02d}", "text": c.text, "start": c.start} for c in children]
hits = search_text(VectorTable.build("children", rows, "id", "text", TOY_A1),
"who resolves a difference between the roof age sources", 3).hits
picked = []
for h in hits:
child = next(r for r in rows if r["id"] == h["id"])
parent = next(p for p in parents if p.start <= child["start"] < p.end)
print(f"2. child {child['id']} ({len(child['text'])} chars) -> parent {parent.rule_id or parent.section}")
if parent not in picked:
picked.append(parent)
print(f" context: {len(picked)} parents, {sum(p.end - p.start for p in picked)} chars; R-ROOF-03 whole: "
f"{any('resolves the difference' in p.text for p in picked)}")
# 3. Temporal: retrieve the rule version that was in force on a given date.
def in_force(rule_id, on):
versions = [p for p in guidance_passages() if p["rule_id"] == rule_id and p["effective"] <= on]
return max(versions, key=lambda p: p["effective"])["passage_id"]
for on, why in (("2025-06-06", "decision date of PRE-023"), (AS_OF, "today in the example")):
print(f"3. R-ROOF-01 in force on {on} ({why}): {in_force('R-ROOF-01', on)}")
1. q0 top 2: ['GD-NOTE-01', 'GD-2026.2-R-ROOF-03'] rank of R-ROOF-03: 2
1. q1 top 2: ['GD-2026.2-R-ROOF-03', 'GD-NOTE-01'] rank of R-ROOF-03: 1
1. q2 top 2: ['GD-NOTE-01', 'GD-2026.2-R-ROOF-03'] rank of R-ROOF-03: 2
fused top 2: ['GD-NOTE-01', 'GD-2026.2-R-ROOF-03'] searches run: 3
2. child C11 (145 chars) -> parent Roof inspection reports
2. child C05 (102 chars) -> parent R-ROOF-03
2. child C04 (127 chars) -> parent R-ROOF-03
context: 2 parents, 599 chars; R-ROOF-03 whole: True
3. R-ROOF-01 in force on 2025-06-06 (decision date of PRE-023): GD-2025.4-R-ROOF-01
3. R-ROOF-01 in force on 2026-11-01 (today in the example): GD-2026.2-R-ROOF-01
Read the three parts as three lessons. Multi-query cost three searches and changed nothing, because the toy encoder already folds these paraphrases; that is the result an evaluation should be able to show before the technique ships. Parent-child turned three small matches into two whole sections and collapsed the duplicate parent, so the model reads all of R-ROOF-03, including who resolves the difference. The as-of filter returned a different passage for each date from the same question. The demonstration skips what production needs: the rewrites come from a model, the parent table is versioned with the children, and the as-of date comes from the application.
Choosing among them#
With each technique explained, the table puts them side by side.
| Technique | Fixes this failing slice | Main cost | Changes the corpus? |
|---|---|---|---|
| Query rewriting | user wording far from corpus wording | a model call per query; drift | no |
| Multi-query | relevant rows right outside the top k | N searches + a model call | no |
| HyDE | short questions vs long formal passages | a generated passage per query; invented text | no |
| Parent-child | small chunks that cut a rule | parent bookkeeping; larger context units | yes |
| Hierarchical | large structured corpora | errors at the top level are final | yes (levels or summaries) |
| Late interaction | single tokens that decide relevance | one vector per token | yes, re-embed |
| Multimodal | evidence in images | a new model and migration | yes, re-embed |
| Graph | relations across unstructured documents | building the graph | yes |
| Temporal | answers that depend on a date | trusted effective dates | metadata only |
| Agentic | questions needing several retrievals | more calls; less predictable | no |
Two patterns stand out. The techniques that do not change the corpus are cheap to try and expensive to run, because they spend model calls on every query. The techniques that change the corpus are expensive to try, because they need re-ingestion and often a full embedding migration as on operations, and cheap per query afterwards.
Common confusion. Late interaction is not reranking, and HyDE is not query rewriting. Late interaction stores many vectors per passage and can score a large candidate set; a cross-encoder reranker reads query and passage together and scores a few dozen. HyDE embeds a generated answer, not a better question. Mixing the terms leads to measuring the wrong thing.
How these techniques fail#
Every technique on this page has a failure that looks like success. A rewrite that drops "wind zone C" retrieves roof rules for every zone, and the answer reads well. A HyDE passage that invents a 20-year limit pulls its neighbors toward that number and, if anyone lets it into the context, becomes a citation of nothing. A parent that was edited while its children were not returns a section that no longer contains the matched sentence. An agentic loop that lets the model choose the tenant filter can retrieve another book's precedents: UW-R07, reached through a tool argument.
The defenses are the same each time. Log the transformed query, the generated text and every tool call in the retrieval trace. Keep filters that encode permissions and versions out of the model's hands. Evaluate on the slice the technique was added for and on the slices it was not, because a technique that helps vague queries can hurt identifier queries. Metric mechanics are on evaluation.
A checklist for Monday#
| Question | Why it matters |
|---|---|
| Which labeled slice fails today, and which technique addresses that slice? | A technique without a failing slice is cost without a measured benefit |
| Is the baseline evaluated with the same queries and labels? | The gain is a difference, not a number |
| Does any generated text (rewrite, HyDE passage) stay out of the evidence? | Invented text must never be cited |
| Do permission and version filters stay outside the model's control? | Agentic and rewritten queries must not widen access |
| Does the technique change the corpus, and is a migration planned? | Index-side techniques mean re-ingestion and often re-embedding |
| Is every transformed query and tool call in the retrieval trace? | Failures here look like successes |
The latency arithmetic of query-side techniquesModel calls before retrieval are on the critical path
Query-side techniques add work in series. A rewrite or a HyDE passage must be generated before the first retrieval can start, so its latency adds directly to every query. Multi-query can run its N searches in parallel, but the slowest of them sets the pace, and the fusion waits for all. Agentic retrieval adds a model turn per retrieval, and the number of turns varies by question.
Three design responses are common. Run the plain query and the transformed one in parallel and fuse them, so the plain query bounds the worst case. Apply the technique only to a routed slice, such as queries that the first retrieval answered with low scores or few rows above a threshold. And set a hard budget on agentic loops: a maximum number of retrieval calls per question, recorded in the trace, after which the agent must answer or abstain. Each of these is measured as p95 latency per stage, not as a mean.
Generated text is never evidenceRewrites, hypothetical passages and summaries must stay out of citations
Several techniques put model-written text into the retrieval path: a rewritten query, a HyDE passage, a summary used by hierarchical retrieval, entity relations extracted for a graph. Each is useful for finding real rows. None of it is a row the insurer wrote, and none of it may be cited as one.
The rule for the grounded answer, on grounding, is that every citation names a retrieved passage or record with its evidence id. Enforce it by construction: generated texts are stored in fields the prompt builder never reads, the trace marks them as generated, and the citation check rejects any id that is not a retrieved row. Summaries for hierarchical retrieval deserve extra care because they look like corpus text; store them in their own table with the version of the text they summarize, and regenerate them when that text changes, as part of the same ingestion job.
Two timelines: when a rule applied, and when we knew itEffective dates versus recorded dates in audits
Temporal retrieval on this page uses one timeline: the effective date of a policy version. An audit sometimes needs a second one: when a record was written into the system. A precedent decided on 2025-06-06 under 2025.4 is judged against the rule effective that day. But if a correction to that precedent was recorded later, a reviewer may ask what the system knew on the day the agent made a recommendation.
Databases have long modeled this as two time columns per version: valid from and to (when it applied in the world) and recorded from and to (when the system held it). A retrieval for an audit then filters on both. Most retrieval systems need only the first timeline, and adding the second is a schema decision to make before the corpus grows. What every system needs is that the date a query uses comes from the application, is written to the trace, and defaults to now.
What late interaction costs in storage, by arithmeticOne vector per token instead of one per passage
With one vector per passage, a corpus of 100,000 passages at 384 dimensions stores 100,000 vectors; with pgvector's documented 4 bytes per dimension plus 8, that is about 154 MB, as the scale page computes. With one vector per token and passages of about 100 tokens, the same corpus stores about ten million vectors. Even at a smaller per-token dimension, that is an order of magnitude more storage and memory for the same text (arithmetic, not a measurement; per-token dimensions and compression vary by model and product).
That is why products that support it often use it where the candidate set is already small. Vespa documents ColBERT tensors for ranking rather than for the approximate retrieval step, which fits the arithmetic: score per-token similarity over the top few hundred candidates found by a single-vector or keyword first stage. Before adopting it, compare its gain against the cheaper alternatives on the same slice: hybrid search for identifiers, a cross-encoder reranker over 50 candidates.
In one breath#
Every advanced technique changes the query, the stored units or the filter, and each has a cost that follows from which one it changes: query-side techniques spend model calls on every query, index-side techniques need re-ingestion and often re-embedding, and temporal retrieval needs trusted dates. None of them should ship without a labeled slice that fails today and an evaluation that shows the fix, and none may let generated text become evidence or let the model choose its own permissions.
The underwriting agent now has a pipeline that can be operated and a catalogue of techniques it can add when a measured failure calls for one. What has not been shown yet is the whole system in one picture: where each corpus lives, which engine runs it, and how a team proves the design works. The next page, architectures, draws four complete designs and gives each a repeatable benchmark recipe.