All chapters
  1. Start hereRetrieval is a query. The model reads only what it returns
  2. Foundations · 1What retrieval means: lookup, keyword search, semantic search
  3. Foundations · 2Traditional SQL first: WHERE, indexes, full-text search and BM25
  4. Foundations · 3Vectors without AI: coordinates, distance and nearest neighbors
  5. Foundations · 4Embeddings: how a model turns text into a vector
  6. Foundations · 5Choosing an embedding model: quality, cost, privacy and fit
  7. Search · 6Schema and storage: where the vectors live
  8. Search · 7Exact vector search: compare the query with every row
  9. Search · 8Approximate nearest neighbors: HNSW, IVF and the recall trade
  10. Search · 9PostgreSQL and pgvector: vectors inside a relational database
  11. Search · 10SQLite vector search: one file on one machine
  12. Pipeline · 11Chunking and ingestion: from a document to retrievable units
  13. Pipeline · 12Retrieval pipelines: from a question to the context the model reads
  14. Pipeline · 13Hybrid search: words and vectors together
  15. Pipeline · 14Reranking: a second, more careful look at the candidates
  16. Pipeline · 15Top-k and scale: ten thousand, a hundred thousand, a million rows
  17. Pipeline · 16Two corpora: similar applicants and underwriter precedents
  18. Pipeline · 17Grounded generation: citations, abstention and conflicts
  19. Pipeline · 18Evaluating retrieval: recall@k, MRR, nDCG and faithfulness
  20. Practice · 19Google Cloud choices: Cloud SQL, AlloyDB, Vector Search, Agent Retrieval, RAG Engine, Agent Search
  21. Practice · 20Other ecosystems: libraries, engines and managed services
  22. Practice · 21Operations: migrations, deletes, permissions and freshness
  23. Practice · 22Advanced retrieval: rewriting, hierarchies, late interaction, graphs, agents
  24. Practice · 23Reference architectures: local-first, dual-corpus, managed, Postgres-native
  25. Practice · 24Framework recipes and the decision guide
  26. ReferenceGlossary
  27. AboutAbout this site and its sources
Pipeline · 11

Chunking and ingestion: from a document to retrievable units

Parsing, sections, size and overlap, metadata, deduplication, versions and document changes. What one retrievable unit is decides what retrieval can ever return.

The unit decides the answer
Retrieval can only return what ingestion stored. A chunk cut through the middle of a rule returns half a rule, however good the search.
Size is not structure
A recursive splitter keeps chunks under a size; a section-aware splitter keeps headings and rules whole. Overlap softens a bad cut; it does not repair one.
Ingestion runs again
Documents change. A content hash per chunk says which vectors to keep, which to re-embed and which to delete, and a document version keeps old policy from posing as current.

Policy version 2026.2 of the insurer's guidance is loaded into the guidance corpus. The ingestion job reads the policy fixture as one markdown document of 2,557 characters, cuts it into pieces of at most 300 characters with 60 characters of overlap, embeds each piece and stores it. Every step succeeds.

Later the underwriting agent works on a submission whose two roof ages disagree, and it searches the guidance: "who resolves a difference between the roof age sources". The nearest piece comes back, characters 246 to 528 of the document. It contains the heading R-ROOF-03 and the condition, and it ends with the words "escalate to a senior underwriter." What it does not contain is the rest of the rule: compare the two ages, record both values and both evidence ids in the recommendation, do not pick one source silently, a senior underwriter resolves the difference. Those sentences are in the next piece, which did not rank first.

This is this site's own case UW-R08. Nothing failed. The search returned the nearest stored unit, exactly as designed. The stored unit was half a rule, and the recommendation built from it can escalate while dropping the instruction to record both evidence ids. The fault entered before any query ran, at the moment the document was cut.

What this is not. This page is about turning documents into retrievable units. Choosing how many of those units fit into the model's context window, and in what order, is context engineering, covered at context.sammuthu.com. Retrieving a small unit and expanding it to its parent section is an advanced technique on the advanced retrieval page.

A chunk is the only thing retrieval can ever return#

Recall the primitive. An embedding column holds one vector per row; a vector query compares the query vector with the rows' vectors, ranks the rows by distance and returns the nearest k; the model reads only what retrieval returned. Retrieval is a query. The model reads only what it returns. For a long document, "the row" is not the document. It is a piece of it.

A chunk is a contiguous piece of a longer document, stored as its own row with its own vector. For example, the R-ROOF-03 section of the policy fixture, characters 401 to 774, can be one chunk. Chunking is the step that decides where those pieces start and end. It matters because it fixes the vocabulary of every later answer: if no chunk holds the whole rule, no query, index or reranker can return the whole rule.

Why cut at all? An embedding is one fixed-length vector for one text. A whole policy guide squeezed into one vector blurs every rule into an average, and the model would then receive the whole guide for every question. Very small pieces have the opposite problem: each one is sharp, but a sentence torn from its rule loses the condition that gives it meaning. Chunking is the trade between those two failures, and it is made once at ingestion, long before anyone asks a question.

Before AI. Search engines faced the same choice as document granularity. A Lucene index over a manual can treat the whole manual as one document, each chapter, or each paragraph; the answer decides what a hit can point to. Database designers make the same call when they normalize: store the order or each order line. A chunk is that decision for text that a model will read.

Parsing comes first: text with its structure#

Before anything can be cut, a file must become text. Parsing is that step, and it is where structure is either kept or lost. A markdown or HTML file carries its headings in the markup. A PDF stores positioned glyphs, so headings, columns, page headers and tables must be reconstructed. A scanned inspection report is an image; optical character recognition (OCR) turns it into text, with the errors that recognition brings: a 9 read as a 0 in a roof age, a table flattened into a line of numbers.

The policy fixture in this example is markdown, and its structure is explicit. The document starts like this (the full document is 2,557 characters, from data.guidance_document(); trimmed at the "..." marks):

# UW-GUIDE 2026.2 (illustrative)

Synthetic policy fixture for teaching. Not insurance practice.

## Roof

### R-ROOF-01

Roof older than 25 years in wind zone C: not eligible. ...
...
### R-ROOF-03

Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter. Compare the age stated by the broker with the age from the roof-data provider or an inspection report. Record both values and both evidence ids in the recommendation. Do not pick one source silently; a senior underwriter resolves the difference.

Two things a parser should hand to the chunker survive in that text: the headings (## Roof, ### R-ROOF-03) and the paragraph breaks. A parser that drops them, for example by extracting a PDF as one long line, leaves the chunker with nothing to respect but character counts. Parsing quality is therefore a retrieval property, and it should be checked like one: open a sample of parsed documents and look at them before embedding anything.

Splitting by size: characters, tokens and a recursive splitter#

The obvious first approach is to cut by size. Choose a maximum, say 1,000, and cut there. Two questions follow at once: 1,000 of what, and where exactly to cut.

Size can be counted in characters or in tokens. A character splitter counts letters and spaces. A token splitter counts the units the embedding model reads; a token is a piece of a word, so a token count tracks what the model sees more closely. The difference matters because embedding models have an input limit and handle longer input by truncating it. As documented on 2026-11-18, the all-MiniLM-L6-v2 model card says input longer than 256 word pieces is truncated by default, and the bge-small-en-v1.5 model card lists a sequence length of 512. A chunk longer than the limit is embedded from its first part only, and the rest of its text is stored but invisible to the vector. The embeddings page covers truncation; for chunking, the rule is that the maximum chunk size must fit the model's limit, measured in its own tokens.

Where to cut is the job of a recursive splitter. It tries a list of separators in order, from the most meaningful to the least: a blank line between paragraphs, then a line break, then a sentence end, then a space, then any character. It splits on the first separator that appears, and only pieces that are still too long are split again with the next separator down. The result keeps paragraphs whole when they fit, sentences whole when paragraphs do not, and falls back to cutting words only as a last resort. These are the separators in examples/rag-uw/chunker.py, unchanged:

# excerpt from examples/rag-uw/chunker.py
SEPARATORS = ["\n\n", "\n", ". ", " ", ""]


def recursive_split(text: str, chunk_size: int = 1000, overlap: int = 200, separators: list[str] | None = None,
                    base: int = 0) -> list[Chunk]:
    """Recursive character splitter with overlap. Offsets are into `text` (+ `base`)."""

The same file adds overlap. Chunk overlap is text repeated at the start of a chunk from the end of the one before, so a sentence that sits near a cut appears in both neighbors. With 1,000 characters and 200 overlap, the last 200 or so characters of chunk 0 open chunk 1. Overlap is a hedge against a bad cut: a fact that straddles the boundary is more likely to be whole in at least one chunk. It costs storage and some duplicate results, and it cannot rescue a unit longer than the overlap, such as a whole rule.

One detail makes every later citation possible. Each chunk in the example carries its start and end offsets, and the file asserts that document[start:end] equals the chunk text. An answer can then cite the exact span of the exact document it came from.

UW-R08, run: where R-ROOF-03 lands#

The obvious approach meets the case here. The script below is a short file added to a scratch copy of examples/rag-uw; it calls only the example's chunker. It cuts the policy fixture with the recursive splitter at 300 / 60, then with the section-aware splitter at the same settings, and reports which chunks hold the rule's heading and its last sentence. A successful run proves where the cuts fall and that every chunk's offsets match the document exactly. It says nothing about any model: no embedding is computed here.

# where_rule_lands.py, run from a scratch copy of examples/rag-uw
import data
from chunker import chunk_containing, recursive_split, section_split

doc = data.guidance_document()          # policy fixture 2026.2 as one markdown document
start = doc.index("### R-ROOF-03")
end = doc.index("resolves the difference.") + len("resolves the difference.")
print(f"document: {len(doc)} characters; R-ROOF-03 spans [{start}:{end}]")

for label, chunks in (("recursive 300/60", recursive_split(doc, 300, 60)),
                      ("section-aware 300/60", section_split(doc, 300, 60))):
    head = chunk_containing(chunks, "### R-ROOF-03")
    tail = chunk_containing(chunks, "resolves the difference.")
    whole = [c.ordinal for c in chunks if c.start <= start and c.end >= end]
    print(f"{label}: {len(chunks)} chunks; heading in {head}; last sentence in {tail}; whole rule in {whole}")
    for c in chunks:
        if c.ordinal in set(head) | set(tail):
            print(f"   chunk {c.ordinal} [{c.start}:{c.end}] ...{c.text[-48:]!r}")
    assert all(doc[c.start:c.end] == c.text for c in chunks)   # offsets point at the exact span
print("every chunk equals document[start:end]: True")
document: 2557 characters; R-ROOF-03 spans [401:774]
recursive 300/60: 12 chunks; heading in [1]; last sentence in [2]; whole rule in []
   chunk 1 [246:528] ...' than 5 years: escalate to a senior underwriter.'
   chunk 2 [529:804] ...'es the difference.\n\n## Sprinklers\n\n### R-SPRK-01'
section-aware 300/60: 18 chunks; heading in [4]; last sentence in [4]; whole rule in [4]
   chunk 4 [401:774] ...'y; a senior underwriter resolves the difference.'
every chunk equals document[start:end]: True

The rule is 373 characters long and the recursive splitter allows 300, so it cannot fit; the splitter did its job and cut at a sentence end. Chunk 1 runs from the end of R-ROOF-02 into the first sentence of R-ROOF-03. Chunk 2 holds the rest of the rule and then, because it fills up to the size, the ## Sprinklers heading and the next rule's heading. Neither chunk is about one thing, and the overlap of 60 characters is far shorter than the rule. In the example's own run of the case (run_demo.py), the query retrieves chunk 1, [246:528], which ends "escalate to a senior underwriter."

A splitter that knows what a rule is#

The improved approach cuts at structure first and size second. A section-aware splitter reads the document's headings, treats each heading-delimited section as a candidate unit, and never cuts inside a rule section, even when that section is longer than the size; in the example it marks such a chunk oversize instead. Sections that are not rules are kept whole when they fit and split recursively inside the section when they do not, so a cut never crosses a heading.

The run above shows the result: 18 chunks instead of 12, because short sections such as ## Roof become their own small units, and chunk 4 is exactly [401:774], the whole of R-ROOF-03 from heading to last sentence. In the example's run, the same query retrieves that chunk. The fix did not touch the search. It changed what was stored.

Diagram: UW-R08: the same span of the policy fixture cut by two splitters

Both lanes cover characters 246 to 804 of the 2,557-character policy fixture. The rule R-ROOF-03 spans 401 to 774. The size-only splitter cuts inside the rule at a sentence end; the section-aware splitter cuts on headings, so one chunk holds the whole rule.

  1. Recursive 300 / 60
    1. Chunk 1 [246:528]: Holds the heading and condition, ends 'escalate to a senior underwriter.'
    2. Chunk 2 [529:804]: Holds 'record both values ... resolves the difference' and the start of the Sprinklers section.
  2. Section-aware 300 / 60
    1. Chunk 3: The previous rule, whole, as its own unit.
    2. Chunk 4 [401:774]: Heading to last sentence in one chunk; the query retrieves it.

The diagram puts the two splitters over the same span of the document. The recursive splitter's boundary falls at the first sentence end after 300 characters, which happens to be inside the rule; the section-aware splitter's boundaries fall on headings. The general lesson: a size limit is a constraint, not a design. The design is the unit you want a query to be able to return, and for policy text that unit is a rule.

Section-aware splitting has costs too. It depends on the parser keeping headings. It produces uneven chunk sizes, so an oversize section can exceed the embedding model's input limit and be truncated: such a section must be split with care (for example, at its sub-headings) or summarized, never cut silently. And it assumes the document's sections match the questions people ask, which is true for a policy fixture and less true for a long narrative report.

The lab: move the cut, watch the rule#

The lab cuts the current policy fixture with either splitter. Set the chunk size (100 to 1,500) and overlap, or pick a recorded setting; a bar map shows the chunks over the document with the R-ROOF-03 span marked, and the query "who resolves a difference between the roof age sources" retrieves the nearest chunk with the toy encoder, compared with the expectation that it holds the whole rule. Toy encoders and synthetic data show how the mechanism works, not how good any model or product is.

The same result without JavaScript, in this example (chunk ordinals from 0):

Splitter, size / overlapChunksR-ROOF-03 heading inIts last sentence inRule whole in one chunk?
recursive 1000 / 200300yes
recursive 500 / 10070 and 11yes (chunk 1)
recursive 300 / 601212no
recursive 200 / 01625no
recursive 150 / 302446no
section-aware 300 / 601844yes
section-aware 1000 / 2001844yes

At recursive 300 / 60 the nearest chunk for the query is [246:528], which ends before "a senior underwriter resolves the difference"; the section-aware splitter returns [401:774], the whole rule.

Read the first rows carefully, because they hold a trap. At 1,000 / 200 the recursive splitter keeps the rule whole, but inside a chunk that also holds R-ROOF-01 and R-ROOF-02: whole, and diluted. At 500 / 100 the overlap happens to carry the full rule into chunk 1. Both are luck, not design: add one sentence to R-ROOF-02 and the boundary moves. The section-aware rows give the same answer at both sizes, because the boundaries come from the document.

Semantic splitting: cutting where the meaning changes#

A third family cuts by meaning. Semantic chunking embeds each sentence (or small group of sentences), compares each with its neighbor, and places a boundary where the similarity drops, on the theory that a drop marks a change of topic. It needs no headings, so it suits text without structure: transcripts, emails, long free-text notes.

It is a design described here, not something the example implements, and its costs follow from the mechanism. It embeds the document sentence by sentence at ingestion, which multiplies the embedding work. Its boundaries depend on the embedding model and on a threshold, so a model change can move every boundary and change every chunk id. And a similarity drop is not a rule boundary: R-ROOF-02 and R-ROOF-03 are both about roof age, so a semantic splitter may well merge them. Where a document has explicit structure, use the structure first.

Metadata travels with every chunk#

A chunk without metadata is text nobody can filter, cite or update. Each stored chunk should carry the facts that later stages need, and they are cheapest to capture at parse time, when the structure is still in view.

FieldExampleUsed by
Source and offsetsUW-GUIDE, [401:774]citations to the exact span
Section and rule idRoof, R-ROOF-03filters, citations, section-aware updates
Document version and status2026.2, currentthe version filter of UW-R03
Trusttrusted: true for the policy fixture; false for a broker attachmenttreating retrieved text as data (UW-005)
Tenantbook-eastthe tenant filter of UW-R07
Content hashsha256 of the chunk textdeduplication and change detection
Embedding modeltoy-a@1the model guard of UW-R06

The table is the contract between ingestion and every later page. The exact search page puts policy_version, status and tenant into WHERE; the grounding page cites by passage and evidence ids; the model guard compares embedding_model. If ingestion does not store a field, no later stage can use it.

A record may already be one chunk#

Not every corpus needs splitting. Corpus A of the example holds applicant profiles, and each profile is already one short, fixed-shape text: a banded summary such as occupancy=office; construction=masonry; area_band=10k-25k; year_built_band=1990-2010; roof_age_band=20-30; wind_zone=C; sprinklered=yes. One record, one chunk, one vector. Cutting it would only lose information.

In the insurance build. In that build, Corpus A embeds a fixed banded template per insured property, for example dwelling=…; sqft_band=2500-4000; year_built_band=1990-2010; …; zip3=…: never raw numbers, never personal data, and versioned with embedding_version. Of 1,140 seeded rows there are 737 distinct texts, and each distinct text is embedded once.

That last fact is deduplication in practice. When two records produce the same text, they produce the same vector, so computing it twice is waste. The question is how to know that two texts are the same without comparing every pair, which is the next section.

Deduplication and change detection by content hash#

A content hash is a short fingerprint computed from the exact bytes of a text, such as its SHA-256 digest. The same text always gives the same hash, and any change, even one character, gives a different one. For example, notes/router-setup.md in the Mnemos-like corpus has a chunk whose hash starts 056755dec40e. Store the hash with each chunk, and two questions become one lookup: has this exact text been embedded before, and has this document changed since the last ingestion?

Documents change, and the policy fixture is the example. Version 2025.4 was replaced by 2026.2. Re-ingesting by deleting everything and embedding again works, but it recomputes vectors that did not change and gives every chunk a new id. The better approach chunks the new version, hashes each chunk, and compares with the stored hashes. The script below does that with the section-aware splitter. A successful run proves that hashes alone identify which chunks to keep, re-embed, add or remove, and that re-ingesting an unchanged document embeds nothing. It does not handle deletes in a vector table or concurrent writers.

# reingest.py, run from a scratch copy of examples/rag-uw
# Decide what to re-embed when a document changes, by comparing content hashes.
import data
from chunker import section_split


def chunk_hashes(doc: str) -> dict[str, str]:
    """section title -> sha256 of the chunk text (section-aware chunks)."""
    return {c.section: data.sha256_hex(c.text) for c in section_split(doc, 1000, 200)}


old_doc = data.guidance_document("2025.4")
new_doc = data.guidance_document("2026.2")
print("same document hash:", data.sha256_hex(old_doc) == data.sha256_hex(new_doc))

old, new = chunk_hashes(old_doc), chunk_hashes(new_doc)
same = [s for s in new if old.get(s) == new[s]]
changed = [s for s in new if s in old and old[s] != new[s]]
added = [s for s in new if s not in old]
removed = [s for s in old if s not in new]
print(f"chunks: {len(new)}; unchanged (keep vectors): {len(same)}; changed (re-embed): {len(changed)}; "
      f"added: {len(added)}; removed: {len(removed)}")
print("changed:", changed)
print("removed:", removed)
print("added:", added[:2], "... and", len(added) - 2, "more")
print("re-ingest the unchanged document again -> chunks to embed:",
      sum(1 for s, h in chunk_hashes(new_doc).items() if new.get(s) != h))
same document hash: False
chunks: 18; unchanged (keep vectors): 8; changed (re-embed): 1; added: 9; removed: 1
changed: ['R-ROOF-01']
removed: ['UW-GUIDE 2025.4 (illustrative)']
added: ['UW-GUIDE 2026.2 (illustrative)', 'Roof inspection reports'] ... and 7 more
re-ingest the unchanged document again -> chunks to embed: 0

The document hash differs, so something changed; the chunk hashes say exactly what. Eight sections are byte-identical and keep their vectors. One rule changed: R-ROOF-01, whose roof limit went from 30 to 25 years. The title section changed because the version number is in its heading. The eight guidance notes exist only in the current fixture, so they are new. Re-ingesting the same document a second time embeds nothing, which is the property that makes a nightly ingestion job safe to run twice.

A document version is the identity of one edition of a source, recorded on every chunk cut from it: here policy_version 2026.2, effective 2026-07-01, and 2025.4 before it. Whether old chunks are deleted or kept is a policy decision, not a technical one. For guidance, this example keeps the superseded rows with status = 'superseded', because a precedent decided under 2025.4 (UW-007) must be explainable against the rule in force at the time. Keeping them is safe only if every search filters on status or version, which is case UW-R03. For private notes, the old version of an edited file usually has no value, and its chunks and vectors should be removed.

Diagram: Re-ingesting a changed document: hash each chunk, embed only what changed

Read left to right. A new version is parsed and cut with the same splitter. Each chunk's content hash is compared with the stored ones. Unchanged chunks keep their vectors, changed and new ones are embedded, and chunks that no longer exist are removed with their vectors, in one change.

  1. Cut
    1. Parse version 2026.2: The new edition becomes text with its structure.
    2. Section-aware split: Cuts at headings so each rule is one chunk.
  2. Compare
    1. Hash each chunk: A fingerprint of the exact text; one changed character changes it.
    2. Diff with stored hashes: Says which chunks to keep, re-embed, add or remove.
  3. Write
    1. Embed changed and new: Only chunks whose hash is new are sent to the embedding model.
    2. Remove what is gone: Rows of chunks that no longer exist are deleted together with their vectors.
Production pitfall. A chunk's hash depends on the splitter as much as on the document. Change the chunk size or switch to the section-aware splitter, and every boundary can move, so every hash changes and the whole corpus is re-embedded. That is a re-embedding caused by the content pipeline, not by the model. Record the splitter and its settings with the chunks, as you record the model.

Mnemos's splitter#

In Mnemos. In the code read, Mnemos chunks with a recursive character splitter of about 1,000 characters with about 200 overlap, trying separators from paragraph to line to sentence to word to character, and embeds each chunk with bge-small-en-v1.5 (384 dimensions). Its vectors live in a vec0 table, and deletes clear that table explicitly.

That is the same design as the example's recursive_split defaults, and it fits Mnemos's corpus: personal notes with little formal structure, where a section-aware splitter would have few headings to use. The SQLite page's lab showed the effect of the size on that kind of corpus: the storm-season question matched the roof note at cosine distance 0.6404 with 1,000-character chunks and 0.3399 with 150-character chunks, in that example. Smaller chunks are not better in general; they were closer for that one question.

How chunking fails, and the test for each#

The failures below all return plausible results. That is why each needs a test written in advance, not a glance at the output.

FailureCase or causeTest
A rule split in halfUW-R08, size-only splittingthe top chunk for the rule's query contains its heading and last sentence
Old policy retrieved as currentUW-R03, version not stored or not filteredno superseded chunk in the context
Text after the model's limit ignoredchunk longer than the input limitno chunk exceeds the limit in model tokens
Deleted document still retrievedre-ingestion adds but never removeschunk count per document matches the current version
Garbled numbersOCR or PDF parsing errorsa sample of parsed pages compared with the originals
An injected instruction treated as policyan untrusted attachment ingested without a trust flag (UW-005)every chunk from an attachment is marked untrusted
In a coding agent. A coding agent chunks a repository too, and source code has structure that a character splitter ignores: functions, classes, modules. Cutting by symbol keeps a function whole, as a section-aware splitter keeps a rule whole. Each chunk carries a file path, a commit or content hash and line offsets, so the agent cites exact lines. After a refactor, changed files are re-chunked by hash and deleted files lose their chunks, or the agent retrieves code that no longer exists.

A checklist for Monday#

QuestionWhy it matters
Does a parsed sample still show headings, paragraphs and tables?The splitter can only respect structure it receives
Does the chunk size fit the embedding model's limit, in its tokens?Text beyond the limit is not in the vector
Is there a test that the most important units (rules, clauses) are whole in one chunk?UW-R08 passes every other check
Does every chunk carry offsets, version, status, trust, tenant, hash and model?Later stages filter and cite with them
Does re-ingesting an unchanged document embed nothing?Then a rerun is safe
Do deleted or replaced documents lose their chunks and vectors?Otherwise they keep being retrieved
Are the splitter and its settings recorded with the chunks?Changing them is a re-embedding
Choosing a chunk size by measurement, not tasteLabeled queries decide; the lab shows why defaults are luck

Chunk size is a parameter of retrieval quality, so it is chosen like one: with labeled queries. Write a set of questions, each with the span of the document that answers it, before any chunking run. For each candidate splitter and size, ingest, run the questions, and count how often the top results contain the whole answering span. That is recall@k with spans as labels; the evaluation page computes it by hand.

Two habits keep the comparison honest. Include the questions that cross boundaries, such as the rule-level question of UW-R08, not only questions answered by one sentence. And slice the results by document type: policy text, notes and reports often prefer different settings, and one global size averages them into a setting that suits none. The chunker lab's table is a small version of this experiment: at 1,000 / 200 and 500 / 100 the recursive splitter passed by luck, and the section-aware splitter passed by construction.

Small units to search, larger units to readDecoupling the retrieval unit from the context unit

The tension in chunking is that small chunks are sharp to search and poor to read, while large chunks are the opposite. One family of designs stops forcing one unit to do both jobs: search over small chunks, then hand the model the larger unit each one belongs to, such as its section. The match is precise, and the model still reads the whole rule.

The cost is in bookkeeping. Every small chunk needs a pointer to its parent, the parent must be stored and versioned with the same care, and the context budget is spent on parents, which are larger. When several small chunks of one parent match, the parent should appear once. The advanced retrieval page treats this as parent-child retrieval. For a policy fixture with explicit sections, the section-aware splitter achieves much of the same effect with one unit.

Idempotent re-ingestion: replace a document as one changeKey by document and hash; delete and insert together

An ingestion job will run twice, crash halfway or overlap with itself. It is safe only if running it again leaves the same state. Three rules give that property. Key chunks by the document id and the content hash, not by position, so an unchanged chunk is recognized and its vector reused. Replace a document's chunks as one transaction: delete the rows of the old version that are no longer present, insert the new ones, and update the version field, together. And never let chunk ids be reused, so an orphaned vector somewhere cannot attach to a new chunk.

Where vectors live outside the chunk table, the replacement has a second side. The SQLite page shows a vec0 table that does not cascade deletes; the storage page shows separate stores that do not share a transaction. The ingestion job owns both sides, and a reconciliation check (every vector has a chunk, every current chunk has a vector) belongs in its tests.

Ingestion is where untrusted text entersMark trust at parse time; never let it be inferred later

Retrieval later treats every chunk the same way: a row with text and a distance. The difference between the insurer's own policy and a broker's attachment must therefore be recorded at ingestion, in a field the later stages cannot miss. In the example, the attachment ATT-SUB-1005-01 carries trusted: false and a source of "broker attachment (untrusted)", and the scripted writer quotes such text in an UNTRUSTED block and treats an instruction inside it as data (UW-005).

Three ingestion rules support that. Assign trust from the source of the file, never from its content. Keep attachments in the corpus scoped to their submission, so they are retrieved only for it. And keep OCR output from scanned documents marked as recognized text, so a reviewer knows that a number in it may be a recognition error. Defense in depth against injected instructions is the subject of guardrails.sammuthu.com.

In one breath#

A chunk is the unit retrieval can return, so chunking decides what any later answer can contain. Parse first and keep the structure; cut by size with a recursive splitter only as a fallback, and cut at headings first when the document has them, because UW-R08 shows that a rule cut in half stays half a rule however good the search. Every chunk carries its offsets, version, status, trust, tenant, model and a content hash, and the hash turns re-ingestion into a precise diff: keep, re-embed, add, remove. Some records, like a banded profile, are already one chunk.

The guidance corpus now holds whole rules with versions and hashes, and the agent's question can find R-ROOF-03 in one chunk. Finding it is one stage of several: the query must be encoded, filtered to the current version, gathered with other candidates, reordered, and cut to the five passages the model reads, and any one of those stages can drop the rule again. The next page, retrieval pipelines, follows UW-004 through every stage with a trace.