Your agent isn't failing because you picked the wrong embedding model. It's failing because someone decided, by reflex, that "retrieval = vector DB" and nobody asked the question again. Retrieval is not a component you install. It's an architectural choice between two paradigms with opposite cost structures, and choosing wrong sentences you to an index that desynchronizes, that leaks data across tenants, and that in production multiplies your latency until it's no longer viable. Make the choice with numbers, not dogma.
Two paradigms, not one component
Classic RAG and agentic search aren't two implementations of the same problem. They're two opposite economic contracts. RAG (chunk → embed → index → retrieve top-k → rerank) precomputes a compressed representation of the corpus and answers in a single pass: cheap, low-latency, but it creates an index that goes stale, that lives somewhere, that breaks semantics when it chunks, and whose recall is measured offline against a query frozen at t=0. Agentic search inverts every term of the trade-off. Zero indexing. The model navigates the corpus at runtime with tools (grep/glob/read, SQL, APIs) and reformulates the search with each read until it has enough context.
The analogy I keep using: RAG is an indexed filing cabinet. Someone read the whole corpus once, wrote a summary card per drawer, sorted them by topic. Lookup is instant and cheap. But edit a document and the card lies until you re-file everything. And to answer "compare case A with case C" you have to trust that the cards, written without knowing your question, captured what you need. Agentic search is an investigator with access to the entire archive: no cards to maintain, reads the current documents, and after reading the first decides which to open next. Slower, more "labor hours" (tokens), but never working off stale information and able to chain leads.
The insight a beginner doesn't have: for corpora that fit (or nearly fit) in the window, and for data where exact matching matters (code, IDs, error strings, proper nouns), agentic search or simply "put everything in the prompt with caching" wins on accuracy and operability. Vector RAG wins on high-volume FAQ, low latency, and enormous corpora that don't fit. The embedding isn't dead. It stopped being the architectural default it was in 2023-2024.
The index is a liability
A beginner reasons in recall@k. An expert asks who owns the index and when it lies. The index is an asset you possess: it desynchronizes (edit a file and it lies until re-indexing, the index lag), it lives somewhere (often a third party, one more copy to leak), and it's an attack surface. The real reason Anthropic pulled embeddings from Claude Code wasn't recall. It was that agentic search "works better, simpler, without the security, privacy, staleness and reliability problems," and that their own codebase was too sensitive to upload to a third-party index. Agentic search reads current bytes at runtime: zero index lag, zero copy to guard.
Why the multi-tenant case is structural, not convenience
With per-tenant data, an index per company is a physical artifact that can leak across tenants if a WHERE org_id is forgotten in the retrieval layer. No index, no artifact to leak. Eliminating the possibility of a failure is stronger than mitigating it. This is general principle, not a claim about any specific deployment. [D]
Lexical beats semantic in identifier domains
Embeddings compress meaning and lose precision on rare tokens. An error code like TS-999, a proper noun, a customer ID, a code symbol, a citation number: those are exactly where the vector space blurs what you need exact. BM25/grep match literally. Hybrid search (dense + sparse, fused with RRF) exists precisely because no vector space captures the lexical well. In code and structured data the lexical dominates, which is why ripgrep wins and why it's the natural substrate of agentic search.
Start with the ~200k-token threshold
Anthropic recommends it explicitly: if the corpus fits under ~200k tokens (~500 pages), don't build RAG. Put it all in the prompt with prompt caching. RAG only wins when the corpus exceeds the window. Building an embedding pipeline for 50 documents is expensive over-engineering. Prompt caching is what makes the "everything in context" strategy viable. It amortizes the repeated corpus and brings Contextual Retrieval preprocessing down to $1.02/M tokens. This is your first design question, and the frontier grows with 1M+ windows.
Contextual Retrieval: the chunk lost its context
Chunking breaks coreference. A chunk saying "margin grew 3%" doesn't know which company or which quarter. Anthropic's technique prepends to each chunk an LLM-generated context that situates it in the document, before embedding and indexing with BM25. It cuts top-20 retrieval failure by 49% (from 5.7% to 2.9%), and 67% (to 1.9%) if you add reranking on top. The structural lesson: the problem was never the embedding model, it was that the chunk lost its context. Agentic search avoids it at the root by reading the whole document on demand. And the trick works outside embeddings too. Persisting a context summary per document helps an agent decide what to read.
If you do keep RAG, reranking is almost always the best cost/benefit improvement before you touch the embedding model: retrieve top-150 cheap candidates (dense + BM25), reorder with a cross-encoder that sees query and document together, return top-20. That's the jump from 49% to 67%. The catch is added latency, which is the dominant constraint in production.
Latency amplifies 83x, so tool speed is viability
The most counterintuitive and most production-critical fact: in search agents, end-to-end latency is magnified by over 83x when mean retrieval latency rises from 0.6s to 4.4s. That's wildly disproportionate — far more than the ~7x rise in retrieval time itself — and the mechanism isn't simple iteration-count multiplication. It's a scheduling cascade: minor retrieval delays cause requests to miss their scheduling windows under FCFS, which evicts their prefix KV-cache (hit rate falls from over 30% to under 21%) and forces costly token recomputation. In RAG, with its ~4 fixed calls, latency is stable. The design implication: a slow search tool isn't "slow," it's systemically unviable in an agentic loop. Local ripgrep isn't an optimization, it's a viability requirement. Budget latency per iteration and cap the number of steps.
GraphRAG: a real capability behind a build-cost cliff
GraphRAG answers a query type vector RAG can't: the global ones ("what are the main themes or risks across the whole corpus?"), by building an entity-relation graph and hierarchical community summaries via Leiden community detection. The cost is construction: it runs every document through an LLM. As an illustrative figure, a third-party analysis estimated ~$33,000 for one specific 5GB legal dataset. That's not GraphRAG's canonical cost, it depends heavily on the corpus. Later variants knocked that down, but with distinct numbers worth not conflating: LazyGraphRAG cuts indexing cost to ~0.1% of full GraphRAG (~1000x); the ~6000x figure that floats around belongs specifically to LightRAG's incremental-update token efficiency. Different techniques, different numbers.
How to actually decide
Start every time with the size question. Fits under the window with caching? Don't build RAG. Doesn't fit, and exact matching matters? Agentic search with fast, bounded tools. Doesn't fit, high-volume, low-latency, semantic? Vector RAG, but hybrid (dense + BM25) plus rerank, never embeddings alone, and add Contextual Retrieval if query volume amortizes the preprocessing. Mixed query difficulty? Route by it: one-shot for direct lookups, the agentic loop only for multi-hop or ambiguous, so you don't pay the 3-10x token tax on every trivial question.
A note on the benchmark claims so you don't overgeneralize. Agentic-RL search results (Search-R1 and its lineage) show gains over static RAG on QA benchmarks specifically. That's a scoped result on those tasks, not a universal "agentic always beats RAG" law. And when you evaluate your own retrieval, score it end-to-end (faithfulness, RAGAS-style, plus trajectory graders for what the agent actually opened), not just offline recall@k against a frozen query. Context relevance is the noisiest dimension for an LLM judge, so don't lean on it alone. The honest summary is that the default flipped, and the size of your corpus tells you most of what you need before you write a line.