Retrieval

Two ways to get evidence into a window.

Pre-inference retrieval runs a similarity search before the model sees the question and pastes the results in. Fast, simple, milliseconds. It lives or dies on the search matching the intent, and when the question is multi-hop it returns a pile of things that are individually plausible and collectively useless.

Just-in-time retrieval hands the agent identifiers — file paths, document ids, queries — and lets it fetch what it needs as it goes. This is progressive disclosure: the agent layers understanding, loading only the subset that turns out to matter. It is precise and it is slower, and it fails badly if the tool contract is unclear, because a confused agent chases dead ends at full token price.

Production systems mostly do both. Stable background gets pre-loaded; the long tail stays behind tools. The split is not a compromise, it is a judgement about which parts of the corpus are worth paying for up front.

Fixed-size splits at a rigid boundary. Fast, predictable, and blind: it cuts tables in half, separates rows from the headers that name their columns, and severs sentences mid-clause. Fine for a baseline, and the source of a startling number of production retrieval bugs.

Recursive splits on the coarsest separator that fits — paragraphs, then lines, then sentences. This is the sensible default. It still has no idea what a table is.

Parent-child decouples the two jobs that chunking is being asked to do at once. Small child chunks are indexed, so matching is precise. The larger parent is what gets returned, so the match arrives with enough context to mean something. Index small, return big.

Auto-merge finishes the thought: when enough children of one parent match, collapse them into the parent rather than spending several slots on fragments of one passage.

You can watch each of these in the lab below, on a corpus built so their failures are visible.

Here is the thing that is easy to say and hard to feel until you have seen it: metadata filtering matters more than everything else combined.

The reason is structural. Filings across companies and years are written in near-identical language. Page sixty says "the Company", not "Northwind Logistics fiscal 2024" — the entity and period live in the header and nowhere else. So a chunk from the middle of a document is, semantically, almost indistinguishable from the same passage in a different company's report. Similarity search cannot separate them because they are not different, in the space it measures.

Filtering on entity and period first removes those candidates before scoring, and no amount of similarity tuning can substitute for it. The published ablation on a multi-agent financial RAG system found accuracy falling from 76.0% to 62.0% when metadata pre-filtering was disabled, with Hit@1 dropping from 0.65 to 0.53.

The lab reproduces that shape on its own corpus, and you should go and break it yourself rather than take the number on trust.