You already know what an AI search agent is — Perplexity, AI Overviews, ChatGPT search, pick one. This isn’t that post. This is what’s actually happening between the query and the answer, the part that decides whether the thing you’re evaluating (or building) is any good.
An AI search agent runs a loop: understand the query, retrieve candidate sources, rerank them for relevance, reason over what’s left, and either answer or go retrieve again. Every product in this category is some version of that loop with different tuning. The differences in tuning are the whole ballgame.
The Pipeline, Stage by Stage
Four stages, roughly, though most production systems blur the lines:
- Query understanding — figure out what’s actually being asked, not just the literal words
- Retrieval — pull candidate documents or chunks from an index
- Reranking — reorder those candidates by actual relevance, not just similarity score
- Reasoning + generation — synthesize an answer, decide if more retrieval is needed, cite sources
Stage 4 is where “agent” starts to mean something. A plain RAG pipeline stops after one retrieval pass. An agent can decide the first pass wasn’t good enough and go again — that loop is the actual distinction people gloss over when they use “RAG” and “agent” interchangeably.
Query Understanding: The Step Before Retrieval Even Starts
Raw user queries are bad retrieval inputs. “best cheap flights to tokyo next month” is not a string you want to embed and search against directly — it’s ambiguous on price, dates, and origin.
Most production systems run query rewriting first: expand abbreviations, resolve pronouns from conversation context, sometimes decompose one query into several sub-queries that get retrieved separately and merged. Multi-hop questions (“compare the pricing models of X and Y”) almost always get split before retrieval, because a single embedding search rarely surfaces good candidates for two different entities at once.
This is where a lot of homegrown agents fall apart, honestly — they skip query decomposition entirely and wonder why retrieval quality is inconsistent on anything but simple lookups.
Retrieval: Dense, Sparse, and Why Hybrid Usually Wins
Dense retrieval embeds your query and documents into vectors, then finds nearest neighbors — usually via an approximate nearest-neighbor index like HNSW, since exact nearest-neighbor search doesn’t scale past a few hundred thousand vectors without unacceptable latency. Dense retrieval is good at semantic matching: “car” and “automobile” land close together even with zero shared words.
Sparse retrieval (BM25 and similar) is the older keyword-matching approach. It’s bad at synonyms but very good at exact terms — model numbers, error codes, proper nouns, anything a dense embedding tends to smear together.
Most serious systems run both and fuse the results, usually with reciprocal rank fusion. Pure dense retrieval alone tends to miss exact-match queries; pure sparse alone misses paraphrases. Hybrid isn’t a nice-to-have at this point, it’s closer to table stakes.
Reranking: The Step Everyone Underbuilds
Initial retrieval — dense, sparse, or hybrid — gets you maybe the top 50-100 candidates cheaply. It does not get you the right top 5. That’s what reranking is for.
A cross-encoder reranker looks at the query and each candidate document together, in the same forward pass, instead of comparing precomputed embeddings independently. That’s expensive — you can’t precompute it, you have to run it per query — which is why it only runs on the shortlist, not the whole index. But it catches relevance signals bi-encoder retrieval structurally can’t see, because a bi-encoder never lets the query and document actually interact before scoring.
Skip this stage and your agent’s answers are only as good as whatever the cheap first-pass retrieval happened to rank first. This is the single most common corner cut in agents that “mostly work but sometimes give weird answers.”
The Agent Loop: ReAct and Tool Calls
This is the part that makes it an agent instead of a pipeline. The common pattern is ReAct — Reason, Act, Observe, repeat:
- Reason: the model decides what it needs next
- Act: it calls a tool — search, a calculator, an API, another retrieval pass
- Observe: it reads the result
- Repeat or answer: either it has enough, or it loops again
Tool calls are structured — the model outputs a function name and arguments matching a defined schema, not free text. That structure is what makes multi-step agents reliable enough to ship; free-text tool invocation is a parsing nightmare and fails silently in ways that are hard to debug.
The hard part isn’t the loop mechanics. It’s teaching the model when to stop looping. Too eager to answer, and it gives a confident wrong answer off thin evidence. Too eager to keep searching, and latency and cost both blow up — every extra hop is another retrieval call, another rerank pass, another model call.
Grounding and Citation Aren’t Decorative
A citation next to a generated sentence isn’t just UX polish — it’s supposed to be a verification mechanism. The generation step is prompted (or fine-tuned) to only assert what it can trace back to a specific retrieved chunk, and the citation is that trace.
In practice this breaks in a specific, predictable way: the model can still generate a plausible-sounding sentence and attach a citation to it that doesn’t actually support the claim. Citation presence and citation correctness are different properties, and most evaluation setups only check the first one. If you’re evaluating an agent’s output quality, spot-check that the cited source actually says what the answer claims it says — don’t just check that a citation exists.
Memory: Mostly Shorter Than People Assume
Context windows keep growing, but most AI search agents don’t carry deep long-term memory across sessions by default — each query effectively starts a fresh retrieval process, sometimes with a summary of prior conversation turns bolted on, not a running memory store.
Where persistent memory does show up, it’s usually implemented the same way as document retrieval: past interactions get embedded and stored in a vector index, then retrieved back in when relevant — memory as a retrieval problem, not a separate subsystem. Worth knowing if you’re evaluating a product on “does it remember what I told it last week.”
Where This Breaks in Production
The failure modes that don’t show up in a demo:
- Retrieval drift — your index goes stale, and the agent confidently retrieves outdated or wrong information because nothing tells it the data is old
- Cost compounding — every additional reasoning hop multiplies model calls, reranker calls, and retrieval calls; a 3-hop query can cost 5-10x a single-hop one
- Latency stacking — same problem, different metric. Users notice a 6-second answer even if it’s correct
- Over-retrieval — pulling in too much context dilutes the reasoning step; more retrieved chunks isn’t automatically better, past a point it’s noise the model has to reason around
- Silent tool-call failures — a malformed function call that fails without a clear error can make the agent proceed on a wrong assumption instead of retrying
None of these show up if you test with five clean queries. They show up in week three of production traffic.
If You’re Building One, Not Just Evaluating One
The architecture above isn’t unique to consumer search products — it’s the same shape as any retrieval-grounded agent, including the ones people build with LangGraph or wire together in n8n for internal tools. Grounding a model properly, before adding agentic behavior on top, tends to be the difference between a demo and something that survives real traffic. And if you’re coordinating multiple tools or retrieval sources, Model Context Protocol is worth understanding before you build your own version of it from scratch.
FAQ
What’s the difference between RAG and an AI search agent?
RAG is retrieve-once-then-generate. An agent can evaluate its own retrieval, decide it’s insufficient, and retrieve again — the loop is the difference, not the underlying retrieval mechanism.
Why rerank if retrieval already returns ranked results?
Because first-pass retrieval (especially pure vector similarity) optimizes for finding candidates fast, not for finding the single most relevant one. Reranking trades speed for precision on a much smaller shortlist.
How does an agent decide when to stop searching and answer?
Usually a combination of a confidence heuristic and a hop limit — most production systems cap reasoning loops at 3-5 hops regardless of confidence, specifically to bound latency and cost.
Does a bigger context window replace the need for retrieval?
No. A larger window means you can stuff in more retrieved content, not that you can skip finding the right content in the first place — irrelevant context in a big window still degrades reasoning quality.
Why do AI search agents still hallucinate even with citations shown?
Because citation attachment and citation accuracy are handled by different, imperfect mechanisms. The model can generate a claim and separately attach a source without a hard guarantee the source actually supports it.
Do I need a vector database, or does keyword search work?
Depends on the query patterns. If exact terms (part numbers, error codes, names) matter, pure vector search underperforms. Hybrid retrieval — vector plus keyword — is the safer default for anything beyond narrow, well-defined domains.
What’s the realistic latency cost of a multi-hop agent?
Each hop typically adds a retrieval call, a rerank pass, and a model call. Two or three hops is common in production; beyond that, most teams see diminishing answer quality against a real cost and latency penalty.
