Artificial Analysis has introduced the 'Search Index,' a benchmark that evaluates search API providers for their effectiveness in supporting AI agents. The benchmark measures performance across three key factors: quality, cost, and speed. The initial test includes providers such as Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave. Each provider is tested using the same model, GPT-5.6 Luna, in a standardized agent setup, with only the search provider varying. The agent runs on Stirrup, an open-source framework from Artificial Analysis, and completes 25 runs per task to search and retrieve web pages. The index combines three equally weighted benchmarks, including DeepSearchQA, which has 900 research questions requiring multiple search queries, and BrowseComp, which tests 200 hard-to-find facts needing multi-step browsing. AA-Omniscience covers 600 questions across six knowledge domains. A tool-free baseline, where the model answers on its own, serves as the comparison point. Without search access, the model scores just 33 points. With search, scores range from 65 to 75. Parallel, Exa, and Firecrawl lead with 75, 74, and 73 points respectively.
Better search quality also lowers total costs. The model uses fewer tokens when it gets good results up front. With Parallel Search (advanced), token use drops by over 40 percent compared to the Basic version. Per-task search costs go up, but total cost comes in lower ($0.084 vs. $0.11). Raw speed per query doesn't always mean faster results overall. Parallel Search (turbo) clocks the shortest response time per query (0.51 seconds vs. 1.03 seconds for Basic), but its lower quality (67 vs. 73) forces the agent to run more passes. Total time per task winds up about the same. Artificial Analysis says Parallel, Firecrawl, and Parallel (turbo) hit the best mix of cost and performance. Other providers can apply to join the benchmark. The full methodology is public.
Artificial Analysis says the benchmark provides a standardized way to evaluate search APIs for AI agents. The company emphasized that the methodology is fully transparent and open for other providers to join. The benchmark aims to help users choose the most effective search API for their AI applications, balancing quality, cost, and speed. Source: thedecoder
