When training search agents, attention goes to the model, the reward design, and the prompt — while the search API is treated as an interchangeable part. A controlled experiment Exa published on May 13 asks a more fundamental question: if you swap the search backend during RL, does the agent end up learning different skills? The answer is yes, and the gap runs through accuracy, training cost, and inference cost. Disclosure first: the study was conducted by Exa, which is one of the two backends being compared. Every number below is vendor-reported, and should be read with that in mind.
A controlled swap: only the search backend
The experiment trained two identically configured agents on Qwen3-4B-Instruct-2507 with LoRA adapters, using Tinker: one searched with Exa, the other with a SERP backend proxying Google results. Both shared the same system prompt, adapted from Search-R1. Every search call returned 5 live web results, each snippet capped at 2,000 characters — and the agent did not know which search variant it had, so behavior could not be biased by expectation.
The reward was a single binary signal at the end of the trajectory. Exact substring matching initially invited reward hacking — agents learned to emit long answers or list multiple alternative answers to game a lucky match — so it was replaced by an LLM grader from SimpleQA, with a -0.25 penalty for trajectories that ran past the context limit. Training used the MuSiQue and HotpotQA multi-hop QA datasets with the Dr. GRPO optimizer, at batch size 64, group size 16, 100 steps, LoRA rank 32, and a 32k context limit. The two agents shared an identical action space; the only difference was which backend executed the search.
The scorecard: ahead on all six benchmarks
Evaluation used pass@k with 200 questions per benchmark and n=200 rollouts per question. Alongside the two training datasets, four out-of-distribution benchmarks were added — 2WikiMultihopQA, FRAMES, BrowseComp, and SimpleQA — with an untrained Qwen3-235B-A22B-Instruct-2507 as the reference model.
The results fall into three layers. First, the Exa-trained 4B beat the SERP-trained 4B on pass@1 across all six benchmarks — 0.839 vs 0.798 on 2WikiMultihopQA, 0.311 vs 0.307 on MuSiQue. The gaps are not always large, but the direction is consistent. Second, and more striking: the 4B Exa agent frequently exceeded the untrained 235B reference — 0.839 vs 0.774 on 2Wiki, 0.767 vs 0.730 on SimpleQA. Third, the exceptions: on FRAMES it scored 0.566 against the 235B’s 0.604, and on BrowseComp everyone sat low, with the highest score in the SERP column belonging to the 235B at 0.055. On the two hardest benchmarks, a 4B model plus RL did not flip the table.
The cost ledger: tokens, search calls, and turns
By step 100, SERP training had consumed 1.89B total tokens against Exa’s 1.58B — 20% more. The more practical number is the equivalence point: the Exa agent matched the SERP-trained agent’s final performance using 0.58B tokens, 69% fewer, while making 62% fewer search calls and taking 58% fewer turns. This is not “a little more accuracy for the same money” — training and inference both got cheaper at once, and fewer calls with shorter trajectories translate directly into lower latency and smaller bills at serving time. Notably, query counts rose after RL (see below) while total tokens still fell — more searching was not waste but more efficient decomposition.
Why: denser reward signals and steadier retrieval
Two mutually reinforcing pieces of evidence support the mechanism. The first is signal density: under the same base policy, Exa search results contained the correct answer 10.7% more often than SERP’s — 36.1% vs 32.6% overall, and 34.0% vs 30.5% on the first call of a rollout. In multi-turn RL, seeing the answer more often on every search makes the per-action reward less sparse; the agent more easily learns which queries produce the next piece of evidence, and that compounds across hundreds of thousands of rollouts into sample efficiency. Under the same base policy, one side simply fed the model denser evidence.
The second is stability. Training issued 476,474 search calls in total (224,462 on Exa, 252,012 on SERP). SERP returned fewer than 5 results on 6.74% of calls (16,991) and zero results on 0.19%; Exa came back short on just 12 calls (0.005%). An empty retrieval dilutes the reward outright — the agent can reason correctly and still find no evidence. Nor is this a Qwen3-4B quirk: the same pattern appeared across 8 open-source models, with Kimi-K2.5 seeing 23.58% of SERP calls return fewer than 5 results, and models that favor quoted phrases and the site: operator failing more often.
What the agent learned: decomposition, not repetition
Query behavior visibly changed after RL: average query counts rose from 1.18 to 2.75 on Exa (1.21 to 3.23 on SERP), and rollouts that issued only a single search call fell from 61% to 17% (Exa) and 12% (SERP). The model did not learn to repeat itself: across 415k multi-call rollouts, the overlap of queries, URLs, and snippets stayed below the same-question ceiling, indicating that agents learned to decompose a question into distinct queries rather than re-asking the same one.
One more finding matters for transfer: after swapping the inference backend, the Exa-trained advantage persisted — the learned search skill transfers — and switching to Exa at inference helped agents from both training sources. The strongest configuration remained Exa at both training and inference.
Limits and open questions
The scope limits deserve an honest statement: a single 4B model, 100 steps, no content fetching (agents saw snippets only, never full pages), a reward that depends on an LLM grader to resist hacking, and two benchmarks — FRAMES and BrowseComp — where the 4B did not pass the 235B reference. And the author is a party to the comparison, so every number is vendor-reported. The authors’ own open questions include whether the gap survives larger models and longer training, how to characterize the learned search behavior, and a stronger harness with content fetching and context pruning.
Builder takeaways
Three judgments you can act on. First, treat the retrieval layer as part of the RL environment, not a peripheral utility: pin it down and monitor it before training, or index and ranking changes will drift your results with no way to attribute the cause. Second, extend evaluation beyond final accuracy with per-search answer-inclusion rate, query counts, context consumption, and empty-return rates — exactly the variables that explained the gap in this experiment. Third, token efficiency is a product metric in itself: equal accuracy on shorter trajectories converts directly into latency and cost, and when two backends benchmark similarly, compare the cost ledger before deciding.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.