There Is No Best RAG. Only The Right Fit.

Why Retrieval Architecture Reaches The Balance Sheet

When a company grounds AI in its own documents, the retrieval layer quietly makes three business decisions at once: how often the answers are right, what each answer costs to produce, and how much risk the organization carries when an answer is wrong. Most companies already have the raw material — millions of files sitting in their databases. What they don’t have is an AI that can find the right ones. Either the data is not referenced at all, or it is retrieved so inefficiently that cost climbs and answers go stale.

This is no longer a niche concern. Grand View Research put the retrieval augmented generation market at roughly $1.2 billion in 2024, growing toward $11 billion by 2030 — a rate near 49 percent per year. Microsoft reports $3.70 returned for every dollar invested in generative AI, a figure cited via Mordor Intelligence. Real money is flowing into this layer, which makes choosing it well a leadership question, not only an engineering one.

What RAG Actually Is

Retrieval augmented generation (RAG), named by Patrick Lewis and colleagues in a 2020 paper, acts like giving an AI a searchable library of your documents to consult before it answers a question. Instead of relying solely on its internal memory, the system first retrieves the relevant facts from your files, then uses them to construct a grounded, accurate response. This keeps the AI honest by citing the exact source of its information, and it makes the system flexible — you can update your documents with new data instantly without needing to rebuild or “retrain” the AI from scratch.

The Default Recipe, And Where It Breaks

The recipe most teams start with is simple: cut documents into chunks, turn each chunk into a vector, find the chunks nearest the question, and paste them into the prompt. For simple lookup questions it works well enough to ship, which is exactly why it spread everywhere.

Then the cracks show. Chunking shreds context, so the model sees fragments instead of complete thoughts. Worse, vector search bets that similarity equals relevance, and that bet fails on the hard questions. A question that requires connecting facts across two documents often shares almost no words with the passage holding the answer, so a nearest-neighbor search never finds it. The plumbing is solid. The retrieval logic underneath it is blunt.

A Dozen Ways To Retrieve Better

The fixes came from every direction, and no team needs all of them. The question is never which approach is most advanced — only which one fits the job.

Five Questions To Ask Before Anyone Writes Code

Choosing well controls accuracy, cost, and risk at the same time. Before a line of code is written, run the use case through five questions.

1. 1What shape is the question? One specific fact, a theme across everything you have, or a multi-step investigation.
2. 2What shape is the data? Small and fixed, constantly changing, long and highly structured, a web of linked entities, or visual and scanned.
3. 3How costly is a wrong answer? In high-stakes work it’s worth paying for a system that checks itself before it speaks.
4. 4How fast, and how cheap, must each answer be? Real-time chat lives on a different budget from overnight analysis.
5. 5Once, or a million times? Questions you’ll answer at scale, on data that keeps changing, justify a system that adapts and learns.

What We Saw Testing This On Real Client Documents

Frameworks are useful, but Arsenal AI forms its recommendations by testing architectures on real client work. In a recent engagement building a document intelligence product for the proposal and procurement space, our engineering team ran a structure-aware, reasoning-based retrieval system built on open-source PageIndex head to head against a tuned vector pipeline. Both systems received the same documents and the same questions, and both were scored by an independent evaluation framework on three measures: whether the right context was retrieved, whether the answer addressed the question, and whether the answer stayed faithful to the evidence.

Whole Sections Beat Fragments

1. 1Whole sections beat fragments. The vector pipeline returned stitched paragraph fragments, while the section-aware system returned complete logical units with their headings intact. On questions that required comparing terms across multiple documents, that difference decided whether the answer held together.
2. 2The negative search test was decisive. Asked which of several documents contained a specific legal disclosure, the section-aware system identified the right document and the right section. The vector pipeline reported that the information was not present, even though it was.
3. 3Faithfulness was consistently at the ceiling on standard operations. And when retrieval did go wrong, the failure showed up plainly in the evaluation scores rather than hiding inside a fluent, confident answer — the failure mode that worries us most in client-facing systems.

One caution, and it is the reason we publish observations rather than leaderboards: these results come from one document set and one workload. We treat them as directional evidence for this class of task — complex analysis over long, structured business documents — not as universal benchmarks. That is also exactly why we evaluate on a client’s own documents before recommending an architecture to them.

The Blind Spot Every Architecture Shares

Lay all of these systems side by side and a shared assumption appears: every one of them builds its index up front, the same way for every question, and then freezes it until someone rebuilds it. None of them learns from the questions it actually receives. The index serving a query asked ten thousand times a day is identical to the index serving a query asked once. The field is pouring real money into architectures that all share this gap — and that gap is what our engineering team set out to close.

LiveRAG: The Index That Learns

LiveRAG is built by Arsenal AI in Pittsburgh, released under the MIT license, and rests on a simple conviction: an index should be answerable to its own use. In one line, it is the RAG that learns its own cache.

On day one it behaves like a strong, ordinary hybrid baseline. But it watches real query traffic, and as certain entities are asked about again and again, it promotes them into precomputed, instantly servable understanding while the long tail stays on the standard path. Within weeks the most common questions answer instantly, and everything else is handled exactly as well as before. What gets promoted is not a saved answer string but a structured, provenance-tracked record: facts tied to their sources, the relationships between them, and a summary derived from those facts. The record is rebuilt the moment a source document changes, so it does not drift, and when two sources genuinely conflict, both are kept and the contradiction is shown rather than hidden.

A trust layer adds the rest of the honesty: it checks every answer for grounding, can withhold a weak one, and treats an honest report that something is not in the corpus as a good answer, so the system never learns to prefer a confident fabrication.

On a held-out sample drawn from three public multi-hop benchmarks (HotpotQA, 2WikiMultiHopQA, and MuSiQue), LiveRAG reached 0.915 retrieval recall, found the bridging evidence on the hardest multi-hop links every time in its top five results with zero additional model calls, and ran at roughly $0.0006 per query end to end. The methodology, including the approaches we tried and rejected, is published alongside the code.

We are equally clear about where LiveRAG is not the tool. Corpus-wide sensemaking fits a graph or a summary tree better. Reasoning over one long, structured document is a job for PageIndex. Simple, high-volume, single-fact traffic performs within noise of plain hybrid retrieval. LiveRAG earns its keep when multi-hop linkage, freshness over changing documents, and traffic-shaped precomputation are what the work requires.

What Leadership Should Take From This

For a technical leader, all of this reduces to two questions worth asking any vendor. The first is universal: how do you measure retrieval quality on our documents, not on your demo? Any serious vendor should have an answer, because a benchmark run on someone else’s corpus says almost nothing about yours. The second question separates the field: does the system get better at our questions over time, and can you show us that number?

Most systems only improve when a human rebuilds the index, so a retrieval system that cannot answer the second question is a snapshot. One that can is an asset that compounds. In fairness, that second question is the one we designed LiveRAG to answer, so weigh our interest accordingly — the difference is that the claim is checkable, since the code, the benchmarks, and the holdout methodology are public.

A retrieval system that cannot get better at your questions over time is a snapshot. One that can is an asset that compounds.

This is also a fair picture of how Arsenal AI works. Our consulting arm and our venture studio share one engineering team in Pittsburgh, so the same people who benchmark retrieval architectures on a client’s documents are the people shipping those architectures into production inside our portfolio companies — and feeding what breaks back into the next build. That loop is where this field guide came from.

Most firms will recommend a retrieval architecture. Arsenal AI has built them, benchmarked them on real client work, and released the one the field was missing.

If you are deciding how to ground AI in your own documents, that is a conversation we have every week — and we would be glad to have it with you.