The precedent corpus of the example started with 40 seeded decisions. This site's own case UW-R10 grows it to 4,000, and the ANN page measured what an index costs and loses at that size. Now the questions get bigger. The insurer plans to load ten years of decided cases, and someone asks the question every retrieval team hears: at 100,000 rows, or a million, do we need a different database? Do we have to re-embed everything? Do we change the top 50?
Each question has a measured answer, and none of them is a row count. UW-R10 already shows the pattern. Going from 40 to 4,000 rows, the example computed vectors for the 3,960 new rows and left the 40 existing vectors byte for byte unchanged. The budgets stayed at 50 candidates, rerank 10, context 5. One number grew a hundredfold: the exact search compared 39 active rows per query before and 3,999 after. Whether that growth justifies an index, a partition or another engine depends on what the queries filter, how often rows change, how many queries arrive at once and how much recall the team can lose. This page shows how to find out.
Retrieval is a query, and the model reads only what it returns. Scale changes how much work the query does to find those rows and how likely it is to miss some. It never changes what the model reads: still the 5 passages the context budget allows.
What this is not. This page does not give a row count at which you need a vector database; nobody can, without your data and traffic. Product capabilities are on ecosystems and Google Cloud. Model migrations, backfills and cutovers are on operations.
What grows with the table, and what stays the same#
Separate the quantities that depend on the number of rows from those that depend on the query. The table below is the summary; the sections after it explain each row.
| Quantity | Grows with rows? | In this example (UW-R10, 40 → 4,000) |
|---|---|---|
| Exact-search work per query | Yes, in proportion | 39 → 3,999 distance computations |
| Index build work and memory | Yes | toy HNSW build grows faster than the rows (harness below) |
Index recall at a fixed ef_search | Tends to fall | ef_search 10: recall@10 0.67 at 4,000 rows |
| Candidates / rerank / context | No: per-query budgets | 50 / 10 / 5 before and after |
| Reranker work | No: follows the pool | 50 candidates scored either way |
| Vectors of existing rows | No | 40 unchanged; 3,960 computed for new rows only |
Read the table in two halves. The top half, search cost and search quality, gets worse as the table grows unless something changes: an index, a setting, a partition. The bottom half, the shape of the answer, does not move. Confusing the two leads teams to scale the wrong thing, for example raising the context budget because the corpus grew.
Common confusion. Growing the table never requires re-embedding. Re-embedding means recomputing vectors, and it is needed only when the vectors would come out different: a new embedding model, a new model version, or a change to the text you embed (a new template, a new chunking rule). Reindexing means rebuilding a search structure from the same vectors. UW-R10 grows the corpus a hundredfold with neither.
The lab shows the bottom half of the table moving, or rather not moving, as you drag the row count from 40 to 40,000 precedents: new vectors only for new rows, zero re-embedded, budgets fixed, exact work growing. Toy encoders and synthetic data show how the mechanism works, not how good any model or product is.
The same numbers without JavaScript (in this example):
| Precedent rows | New vectors computed | Existing rows re-embedded | Candidates / rerank / context | Exact-search distance computations per query |
|---|---|---|---|---|
| 40 | 0 | 0 | 50 / 10 / 5 | 40 |
| 400 | 360 | 0 | 50 / 10 / 5 | 400 |
| 4,000 | 3,960 | 0 | 50 / 10 / 5 | 4,000 |
| 40,000 | 39,960 | 0 | 50 / 10 / 5 | 40,000 |
Ten thousand, a hundred thousand, a million: the work of exact search#
Exact search compares the query vector with every row that passes the filters, so its work is the number of those rows. At 10,000 rows that is 10,000 distance computations per query; at a million, a million. Each computation on a 384-dimensional vector is 384 multiplications and additions, so 10,000 rows are about 3.8 million of them per query (arithmetic). The intuition that this is too slow is often wrong: a sequential scan of a table that fits in memory is work databases are built for, and the exact search page shows the plan.
The work that matters is per query times queries per second. A corpus of 100,000 rows queried once a minute and the same corpus queried 200 times a second are different problems at the same size. That is why the eight-step playbook below measures concurrency and tail latency, not only size. A row count alone is not a requirement.
Before AI. The same reasoning has always applied to plain SQL. A query without a usable index scans the table, and whether that matters depends on table size, query rate and the latency target, not on a magic row count. Teams learned to read EXPLAIN, measure under load and add an index when the numbers said so. Vector search is the same discipline with a different distance.
In the insurance build. That build's documentation reports what the PostgreSQL planner chose on its applicant-profile table: with the selective structured filter, a btree and an exact sort over about 190 rows; unfiltered at 1,140 rows, a sequential scan; the HNSW walk appeared only with enable_seqscan = off. Its precedent table has no ANN index at all, a btree plus an exact distance sort, with HNSW noted in the schema as worth considering only past about 50,000 rows per provision. That note is a design judgment for that build's filters, reported in that build; it is not a threshold for anyone else.
Candidate recall: what an index loses as the table grows#
An ANN index, explained on the ANN page, compares only a share of the rows and may miss true neighbors. Its index recall is the share of the exact top k it returned. At a fixed search setting, that share tends to fall as the table grows: the beam explores about the same number of nodes while the number of near neighbors competing for the top k grows. UW-R10 shows it on the precedent vectors. At 4,000 rows, a toy HNSW at ef_search 10 found 67% of the exact top ten; at 40 it found 93%; at 100, all of them.
Candidate recall matters more than index recall alone because of what comes next. The reranker sees only the candidate pool. If the index loses a relevant precedent from the top 50, no reranker and no prompt can bring it back. So at scale the question "is the index good enough" becomes "is the relevant row still in the top 50", and that is answered with labeled queries, recall@50 against human labels, computed as on the evaluation page.
The lab's scale preset sweeps the same comparison over 16-dimensional toy points at 1,000 and 5,000 rows: exact scan, HNSW at three ef_search values, IVF at three nprobe values, each with its mean work and recall.
The same numbers without JavaScript (16 dimensions, 10 queries, mean distance computations per query with index recall@10 in brackets; in this example):
| Rows | Exact scan | HNSW ef 10 | HNSW ef 40 | HNSW ef 100 | IVF nprobe 1 | IVF nprobe 4 | IVF nprobe 16 |
|---|---|---|---|---|---|---|---|
| 1,000 | 1,000 (recall 1) | 103.8 (0.99) | 147.6 (1.00) | 158.5 (1.00) | 86.4 (0.62) | 166.1 (0.94) | 505.2 (1.00) |
| 5,000 | 5,000 (recall 1) | 176.4 (0.87) | 330.8 (1.00) | 501.2 (1.00) | 220.1 (0.52) | 628.2 (0.95) | 2,067.4 (1.00) |
Read across the HNSW columns. Five times the rows cost about 1.7 times the work at ef_search 10 and lose 12 points of recall; at ef_search 40 recall holds and the work roughly doubles. The index still does a small fraction of the exact scan's work. The price of scale shows up as a setting you must raise and re-measure, not as a cliff.
Filters and selectivity: the number that decides the plan#
Selectivity is the share of rows a filter keeps. The filter wind_zone = 'C' AND tenant = 'book-east' on Corpus A keeps 82 of 240 profiles in UW-R04, a selectivity of about 34%. A filter on one tenant of many, or one provision among dozens, can keep 1% or less. At scale, selectivity matters more than table size, because it decides which plan is cheap.
Two cases pull in opposite directions. With a selective filter, exact search over the filtered rows is cheap: a btree finds them, and the database sorts a few hundred distances. The example's Q13, on a 20,000-row toy table in PostgreSQL, shows the planner choosing exactly that for a filter that keeps 200 rows: a bitmap index scan on the filter column and an exact top-N sort. An approximate index with the same filter does badly, because it applies the filter after the walk: in Q12 the HNSW query returned 0 rows with iterative scan off and 10 rows with hnsw.iterative_scan = relaxed_order (one run on one laptop, toy data; not a benchmark; the pgvector page has the full output). That is UW-R09, post-filter starvation.
With an unselective filter, the opposite holds. Exact search over 80% of a million rows is 800,000 computations per query, and the index walk with a filter that removes few rows loses little.
The harness further down measures the selective case at three sizes: a filter that keeps about 5% of rows. Exact search over the filtered rows costs 54, 251 and 508 computations at 1,000, 5,000 and 10,000 rows, always with recall 1.0. Extend the same 5% to a million rows by arithmetic and it is about 50,000 computations per query, at which point an index that respects the filter, or a partition per filter value, may become the cheaper plan. That crossover depends on your selectivity, your vector length and your engine. It has to be measured.
pgvector's README, read on 2026-11-18, names the structural fixes: for a few distinct filter values, partial indexes (one index per value); for many values, partitioning.
Partitions and shards: splitting the table#
Partitioning splits one logical table into pieces by a key, inside one database. A precedent table partitioned by tenant stores each book of business in its own partition, each with its own indexes; a query that filters on one tenant searches only that partition. Partitioning is the structural form of a selective filter: the planner skips whole partitions before any distance is computed. It also gives operations a smaller unit to rebuild or archive. pgvector's README notes that a non-partitioned PostgreSQL table is limited to 32 TB by default, and that partitioned tables can have thousands of such partitions.
Sharding splits the data across machines. Each shard holds part of the rows and answers part of the query; a coordinator sends the query to every shard, each returns its own top k, and the coordinator merges them into the global top k. Two consequences follow. Each shard must return the full k, because the global top 50 may all live on one shard. And recall now has two places to be lost: inside each shard's index, and in the merge if a shard returns fewer than k. pgvector's README lists replicas for horizontal scaling of reads and Citus, PgDog or another approach for sharding, as documented on 2026-11-18.
Diagram: Partition or shard: skipped by the filter, or asked and merged
Read left to right in each lane. A partitioned precedent table sits in one database, split by tenant; a query that filters on one tenant searches only that partition and the planner skips the others. A sharded table spreads rows across machines; a coordinator asks every shard, each returns its own top k, and the coordinator merges them into the global top k. The design is illustrative; nothing here was measured.
- Partitioning: one database
- Query with tenant filter: The filter names the partition key, so most of the table is excluded up front.
- One partition searched: Other partitions are skipped before any distance is computed.
- Sharding: several machines
- Every shard asked: Each shard must return the full k; the global top k may live on one shard.
- Coordinator merges: The merge is only as complete as the shard lists it receives.
The diagram contrasts the two splits for the precedent corpus. A partition is skipped by the planner when the filter excludes it; a shard is always asked, and its answer is merged. Partition by what your queries filter on. Shard when one machine can no longer hold the data or serve the traffic, which is a measured condition, not a size.
HNSW parameters at scale, and memory by arithmetic#
The three HNSW settings, explained on ANN, keep their meaning as the table grows. pgvector's defaults, as documented on 2026-11-18, are m 16, ef_construction 64 and hnsw.ef_search 40; the insurance build uses m = 16, ef_construction = 64 and hnsw.ef_search = 80 on its applicant-profile corpus.
One pgvector behavior bites as soon as the candidate pool grows: the README notes that results are limited by hnsw.ef_search. An ORDER BY ... LIMIT 50 query with the default ef_search of 40 can return at most 40 rows. A candidate pool of 50 needs ef_search of at least 50, and the harness shows that a larger table needs a larger setting for the same recall anyway.
Memory is arithmetic before it is a measurement. The rows below use two documented figures: pgvector's vector type takes 4 bytes per dimension plus 8, and halfvec 2 bytes per dimension plus 8; hnswlib estimates graph memory at about 8 to 10 bytes per M per element. For 384 dimensions and M = 16:
| Rows | Vectors as vector(384) (1,544 bytes each) | As halfvec(384) (776 bytes each) | HNSW links at M = 16 (128 to 160 bytes each) |
|---|---|---|---|
| 10,000 | about 15 MB | about 8 MB | about 1.3 to 1.6 MB |
| 100,000 | about 154 MB | about 78 MB | about 13 to 16 MB |
| 1,000,000 | about 1.5 GB | about 0.8 GB | about 128 to 160 MB |
These are calculations from documented figures, not measurements; an engine adds row headers, page overhead, the table's other columns and its own index structures. Two conclusions survive the overheads. The vectors, not the links, dominate memory, so the lever is the vector type or quantization (the ANN page explains the trade), not M. And build memory is a planning number: pgvector documents much faster HNSW builds while the graph fits in maintenance_work_mem, raises a notice when it no longer fits, and builds in parallel with max_parallel_maintenance_workers.
Updates at scale: inserts, deletes and rebuilds#
A growing table is also a changing one. An insert into an HNSW index runs a beam search at ef_construction to find the new point's neighbors, so inserts cost more than in a plain table; pgvector's README says higher ef_construction slows inserts. For a bulk load it recommends COPY and adding indexes after the initial data, and CREATE INDEX CONCURRENTLY in production to avoid blocking writes. Deletes leave dead rows that vacuum must clean, and the README notes that vacuuming HNSW indexes can take a while and suggests REINDEX INDEX CONCURRENTLY first.
Each of these is a reindex question, never a re-embedding question. The insurance build keeps the two apart in its schema: stale rows are found by embedding_version and re-embedded in batches; REINDEX INDEX CONCURRENTLY rebuilds the graph; a model change is a new column, a backfill and a switch of reads. In UW-R10's terms: 4,000 rows need no re-embedding; a new embedding model would need 4,000 new vectors whatever the size, and a migration the operations page walks through.
The scaling playbook, in eight steps#
The site's specification sets a playbook instead of thresholds: benchmark against data distribution, update rate, filtering, query concurrency, p95 latency, recall@k, memory and operational cost. The eight steps, each applied to the insurance corpora:
- Start with exact cosine search on a small set and build labeled ground truth. Exact search returns the true top k, so it is the answer key for every later step. Label which precedents and passages are relevant for a few dozen real queries, as the example's
GUIDANCE_QRELSandPRECEDENT_QRELSdo. - Measure recall@50 for profile and precedent slices, and read the query plans. Slice by the filters queries carry: wind zone, tenant, provision. Read
EXPLAINto see which plan ran; the insurance build found the planner preferred a btree and an exact sort. - Introduce HNSW and tune search effort, selectivity and candidate counts against exact results. Sweep
ef_search, filters and pool size; compare each run with exact search (index recall) and with labels (recall@50). Keepef_searchat least the pool size. - Retrieve a broad candidate set, then rerank the top N, and evaluate accuracy against latency. The reranking page's funnel: pool 50 or 100, rerank 10. Measure the reranker's share of the latency.
- Add hybrid search for identifiers, policy text and rare terms. The hybrid page's
UW-R02: identifier queries need the keyword leg. - Test inserts, deletes, policy revisions and index maintenance, independently of the embedding model. Load new precedents, revoke some, publish a policy version, rebuild an index; re-measure recall after each. None of these needs a new model.
- Re-embed and migrate only when model quality, version incompatibility, a changed content representation or domain evaluation justifies it. And then with a second column, a backfill, a dual-index cutover and a rollback path, as on operations.
- Choose PostgreSQL, SQLite, a managed Google Cloud service or a dedicated vector service by measured requirements and governance, not fashion. The measurements of steps 1 to 7 are the input; architectures shows four complete designs that made this choice.
Diagram: The scaling playbook: measure first, change one thing, measure again
Read left to right. Exact search on a small set gives labeled ground truth and an answer key. Every later change (an index and its settings, a broader pool with a reranker, a keyword leg, inserts and policy revisions) is measured against it. Re-embedding and the choice of engine come last, and only on measured reasons.
- Steps 1 and 2: the answer key
- Exact search and labels: The true top k and human labels for real queries.
- Steps 3 to 6: one change at a time
- Index, pool, rerank, hybrid: Tune search effort, filters and pool size; add a reranker and a keyword leg.
- Inserts, deletes, revisions: Re-measure recall after churn; rebuild with the same vectors when it falls.
- Steps 7 and 8: only on evidence
- Re-embed if justified: A second column, a backfill, a cutover and a rollback path.
- Choose the engine: PostgreSQL, SQLite, managed or dedicated, by the numbers above.
The diagram compresses the eight steps into three groups. The left group produces the answer key; the middle group changes one thing at a time against it; the right group spends the evidence. Steps 1 to 3 are where a benchmark harness lives. Step 8 is where its results are spent.
A benchmark harness, built from the example#
A benchmark harness is a repeatable program that runs a fixed query set against each configuration, computes the answer key with exact search, and records recall, work and short results for every setting in one table. Repeatable means the same data, the same queries and the same seeds every run, so a change in the numbers means a change in the system.
The sketch below is built from the example's ann.py. It runs 20 queries against 1,000, 5,000 and 10,000 toy points in 16 dimensions: exact scan, a toy HNSW at three ef_search values, and a 5% filter handled three ways (post-filter, iterative scan, exact search over the filtered rows). It counts distance computations, never milliseconds. A successful run shows how recall and work move with table size and filter selectivity in this toy; it does not predict any real engine, and its numbers are one run on one laptop, toy data, not a benchmark. A real harness replaces the toy index with your database, the toy points with your vectors, the generated queries with logged queries and labels, and adds timing under concurrency.
# bench_harness.py, run from a scratch copy of examples/rag-uw
# A benchmark harness in miniature: same queries, exact search as the answer key, one row per setting.
# It counts distance computations (work), never milliseconds: toy Python timings are not a benchmark.
from ann import HNSW, exact_knn, index_recall, make_categories, points_and_queries
from vecmath import cosine_distance
K, DIM, QUERIES = 10, 16, 20
print(f"{'rows':>6} {'setting':<30} {'recall@10':>9} {'work/query':>10} {'short':>5}")
for n in (1000, 5000, 10000):
pts, qs = points_and_queries(n, QUERIES, DIM, seed=7, decimals=6)
cats = make_categories(n)
selective = lambda i: cats[i] < 5 # about 5% of rows pass the filter
h = HNSW(pts, M=8, ef_construction=32, distance=cosine_distance)
print(f"{n:>6} {'build (M=8, ef_construction=32)':<30} {'':>9} {h.build_computations:>10} {'':>5}")
print(f"{n:>6} {'exact scan':<30} {1.0:>9.2f} {n:>10} {0:>5}")
for ef in (10, 40, 100):
r = index_recall(lambda q: h.search(q, K, ef), pts, qs, K, cosine_distance)
print(f"{n:>6} {'HNSW ef_search=' + str(ef):<30} {r['recall']:>9.2f} {r['distance_computations']:>10.0f} {r['short_results']:>5}")
r = index_recall(lambda q: h.search(q, K, 40, flt=selective), pts, qs, K, cosine_distance, flt=selective)
print(f"{n:>6} {'5% filter, post-filter, ef=40':<30} {r['recall']:>9.2f} {r['distance_computations']:>10.0f} {r['short_results']:>5}")
r = index_recall(lambda q: h.search(q, K, 40, flt=selective, iterative=True), pts, qs, K, cosine_distance, flt=selective)
print(f"{n:>6} {'5% filter, iterative, ef=40':<30} {r['recall']:>9.2f} {r['distance_computations']:>10.0f} {r['short_results']:>5}")
passing = sum(selective(i) for i in range(n))
print(f"{n:>6} {'5% filter, exact over filtered':<30} {1.0:>9.2f} {passing:>10} {0:>5}")
rows setting recall@10 work/query short
1000 build (M=8, ef_construction=32) 380624
1000 exact scan 1.00 1000 0
1000 HNSW ef_search=10 0.99 101 0
1000 HNSW ef_search=40 1.00 146 0
1000 HNSW ef_search=100 1.00 156 0
1000 5% filter, post-filter, ef=40 0.24 146 20
1000 5% filter, iterative, ef=40 0.99 926 0
1000 5% filter, exact over filtered 1.00 54 0
5000 build (M=8, ef_construction=32) 2815559
5000 exact scan 1.00 5000 0
5000 HNSW ef_search=10 0.84 166 0
5000 HNSW ef_search=40 1.00 322 0
5000 HNSW ef_search=100 1.00 494 0
5000 5% filter, post-filter, ef=40 0.22 322 20
5000 5% filter, iterative, ef=40 0.97 1739 0
5000 5% filter, exact over filtered 1.00 251 0
10000 build (M=8, ef_construction=32) 6503720
10000 exact scan 1.00 10000 0
10000 HNSW ef_search=10 0.81 169 0
10000 HNSW ef_search=40 0.94 369 0
10000 HNSW ef_search=100 0.95 630 0
10000 5% filter, post-filter, ef=40 0.25 369 20
10000 5% filter, iterative, ef=40 0.94 2346 0
10000 5% filter, exact over filtered 1.00 508 0
Four readings. First, build work grows faster than the rows: ten times the rows cost about 17 times the build computations (380,624 to 6,503,720) in this toy. Second, recall at a fixed ef_search falls with size, 0.99 to 0.81 at ef_search 10, and at 10,000 rows even ef_search 100 reaches only 0.95: the toy graph with M 8 is likely too sparse for that size, the signal to try a rebuild with a larger M or ef_construction, a reindex with the same vectors. Third, the post-filter rows show starvation at every size: all 20 queries returned fewer than 10 rows, so recall stays near a quarter however big the table is. Fourth, for that 5% filter, exact search over the filtered rows did less work than the iterative scan at every size here (508 against 2,346 computations at 10,000 rows). Each reading is about this toy; the harness is the part to copy.
Production pitfall. A harness that reports a mean hides the slow queries that users feel. Latency is measured as a percentile: p95 latency is the time within which 95% of queries finish, so one query in twenty is slower. Selective filters, cold caches and large candidate pools all live in that tail. Measure p95 at the query rate you expect, on the engine and hardware you will run, and report it next to recall; a fast mean with a slow tail is a slow system.
In a coding agent. A coding agent's repository index grows with every commit, and the same separation applies. More files mean more chunks to embed (only the new ones) and more work per exact search; the agent's budget of retrieved chunks per question stays the same. A monorepo is often partitioned by package, so a query scoped to one package searches only its partition.
What the products document about scale#
Beyond the levers already cited, pgvector's README (read on 2026-11-18) documents filters applied after the approximate scan, with the example that 10% matching rows and ef_search 40 give about 4 rows; iterative scans (strict_order or relaxed_order, off by default) bounded by hnsw.max_scan_tuples, default 20,000; and EXPLAIN (ANALYZE, BUFFERS) for debugging. The HNSW paper by Malkov and Yashunin reports logarithmic complexity scaling for its search. Engine-by-engine scaling options, including disk-based and streaming indexes, are on ecosystems; Google Cloud's managed options on Google Cloud.
A checklist for Monday#
| Question | Why it matters |
|---|---|
| Is there a labeled query set and an exact-search answer key? | Every scaling decision is measured against them |
| Do you know the selectivity of each common filter? | It decides between exact search, an index and a partition |
Is ef_search at least the candidate pool, and re-measured after growth? | pgvector returns at most ef_search rows; recall falls with size |
| Is recall@50 reported per slice, next to p95 latency? | Averages hide the filtered and slow queries |
| Is memory planned by arithmetic before the build? | Vectors dominate; the vector type is the lever |
| Are reindex and re-embed separate runbooks? | Growth needs neither by itself; a model change needs re-embedding |
| Does any document say "N rows needs a vector database"? | Replace it with a measurement |
Scatter, gather and the recall of a mergeWhy every shard must return the full k, and what the coordinator cannot fix
A sharded vector search sends the query to every shard, and each returns its own top k; the coordinator merges by distance and keeps the global top k. The merge is exact if every shard's list is exact. Two things break that. If a shard uses an approximate index, its list may miss a true neighbor, and the merged result inherits the miss. If a shard returns fewer than k, because its filter starved its index (UW-R09 on one shard), the coordinator cannot tell a missing neighbor from a shard that had none.
So the harness of a sharded system measures recall end to end, against an exact answer key computed over all shards, and records short results per shard. The cost side is also different: every query touches every shard, so the slowest shard sets the latency, and the tail of the slowest shard sets the system's p95. Partitioning by the most common filter, where a query touches only its partition, avoids both problems for queries that carry that filter.
What a re-embedding costs at a million rowsArithmetic that explains why teams avoid unnecessary migrations
Re-embedding recomputes one vector per row, so its cost grows with the table in a way search cost does not. At a million rows it is a million calls to the embedding model (or a million forward passes of a local model), a new column or table of about 1.5 GB of vector(384) values by the arithmetic above, a new index built over them, an evaluation of the new space against labels, and a cutover with a rollback path. During the backfill, the old and the new vectors exist side by side, so storage roughly doubles for a while.
That is why growth must never trigger it by accident, and why UW-R10 checks that the 40 existing vectors are unchanged after the table grows. A re-embedding is justified by a measured reason (step 7 of the playbook): a better model on your labeled queries, a model version that is being retired, a changed embedded text. The insurance build records embedding_version per row precisely so that stale rows can be found and re-embedded in batches, and treats a model change as a new column, a backfill and a switch of reads.
Concurrency, caches and the tailWhy a single-query benchmark understates production latency
A harness that runs one query at a time measures work. Production runs many at once, and they compete for CPU, memory bandwidth and cache. An HNSW walk touches nodes scattered across memory; when the graph and vectors fit in memory, a walk is fast; when they do not, each step can wait on storage, and the tail grows. A cold cache after a restart or a failover makes the first minutes slow for the same reason.
Three habits make the measurement honest. Run the harness at the expected query rate and at a multiple of it, and report p95 and p99, not the mean. Warm the cache the way production does, and also measure cold. And separate the stages in the trace, first stage, reranker, model call, so a slow percentile points at a stage. The retrieval pipelines page defines the trace; observe.sammuthu.com treats traces as a discipline.
When exact search remains the right design at scaleSmall filtered sets inside a large table
A large table does not mean large searches. The precedent corpus of the insurance build is queried per provision, with a status filter and a 36-month lookback, so each query compares a small set however large the whole table becomes. That build uses a btree on (provision_key, status, decided_at DESC) and an exact distance sort, and notes HNSW as worth considering only past about 50,000 rows per provision. The number in that note is per provision, not per table, which is the point: the size that matters is the size after the filter.
Exact search over a filtered set has properties an index lacks. Its recall is 1.0 by construction, so there is nothing to tune and nothing to drift; inserts and deletes cost no graph maintenance; and the result is the answer key itself. The harness above shows the same thing in miniature: for a 5% filter, exact search over the filtered rows was the cheapest correct plan at every size measured. Keep exact search until the measured work per query, at your query rate, misses the latency target.
In one breath#
As a table grows, exact-search work, index build cost, index memory and the recall an index loses at a fixed setting all grow; the per-query budgets of 50, 10 and 5 and the vectors of existing rows do not. Selectivity decides the plan more than size does: a selective filter favors exact search over the filtered rows or a partition, an unselective one favors an index tuned against exact results. The eight-step playbook and a harness with exact search as its answer key replace every row-count threshold, and growth alone calls for a reindex at most, never a re-embedding.
The precedent corpus can now grow from 40 rows to 4,000 or a million with its recall measured at each step. One question has stayed in the background: the agent searches two different corpora, similar applicants and decided precedents, with different text, filters, authority and freshness. The next page, two corpora, takes them apart and decides whether they belong in two tables in one database or in two systems.