RAG was all the rage in the early days of widespread AI use, but more and more people seem to say that there might be better alternatives around.
Boris Cherny, a Lead Engineer at Anthropic, the company behind the AI assistant Claude, recently shed light on their journey with information retrieval systems for code generation tasks. In a candid discussion, Cherny detailed why Anthropic transitioned from the popular Retrieval Augmented Generation (RAG) approach to a more dynamic “agentic search” methodology, citing surprising performance gains and significant operational advantages.
The challenge, as Cherny put it, often boils down to efficiently getting relevant information into the AI’s context. “A related problem to memory is how to get information into the context or knowledge base,” he explained. “Originally, very early versions of Claude actually used RAG. We indexed the code base, I think using Voyage, just off-the-shelf RAG, and that worked pretty well. We tried a few different versions of it: there was RAG, and then we tried a few different kinds of search tools. Eventually, we landed on just agentic search as the way to do things.”
The decision to move away from RAG, a technique that involves retrieving relevant documents from a knowledge base and providing them to a large language model as context, wasn’t taken lightly. Cherny highlighted compelling reasons for the shift: “There were two, maybe three, big reasons for this. One is that it outperformed everything else by a lot. By a lot. This was surprising.”
Interestingly, the initial success metric wasn’t a formal benchmark. When asked about the specific benchmark that demonstrated this significant outperformance, Cherny candidly stated, “This was just ‘vibes’ – internal vibes. There are some internal benchmarks also, but mostly vibes. It just felt better,” he said.
So, what exactly is this “agentic search” that Anthropic found so effective? Cherny clarified, “What does agentic search mean in this context? It means letting the agent look up information in however many search cycles it needs, just using regular code searching tools like glob and grep – regular code search.” This approach empowers the AI model to actively seek out information using standard developer tools, iteratively refining its understanding and output.
Beyond the sheer performance uplift, practical considerations around RAG’s indexing process played a crucial role in Anthropic’s decision. “The second big reason was the whole indexing step required for RAG,” Cherny elaborated. “There’s a lot of complexity that comes with that because the code drifts out of sync. Then there are security issues because this index has to live somewhere. What if that provider gets hacked? It’s just a lot of liability for a company to do that.”
He emphasized the sensitivity of their own data: “Even for our codebase, which is very sensitive, we don’t want to upload it to a third-party thing. It could be a first-party thing, but then we still have this out-of-sync issue. Agent search just sidesteps all of that.”
The trade-off, however, is resource consumption. “So essentially, at the cost of latency and tokens, you now have really awesome search without the security downsides,” Cherny concluded.
Implications of Shifting from RAG to Agentic Search
Anthropic’s experience offers valuable insights for the broader tech and programming community. The move from a relatively static retrieval system like RAG to a dynamic, tool-using agentic approach for code generation suggests a potential paradigm shift. While RAG has proven effective for many applications, its reliance on pre-indexed, potentially stale data can be a drawback, especially in rapidly evolving environments like codebases.
The “vibes” metric, while informal, is also telling. It underscores that developer experience and the perceived quality of AI assistance can be powerful drivers for architectural decisions, sometimes outpacing or complementing formal benchmarks. When a system “just feels better” and demonstrably improves workflows, it warrants serious consideration. The fact that agentic search outperformed RAG “by a lot” is a significant claim that could prompt other organizations to explore similar methods.
Furthermore, the security and operational concerns raised by Cherny are critical for any business handling sensitive information. The complexity of maintaining an up-to-date, secure index for RAG is a tangible overhead. For companies dealing with proprietary code or confidential data, the agentic model’s ability to search live, existing systems using standard, auditable tools offers a compelling advantage, even if it comes at the cost of increased latency and token usage. This aligns with a growing emphasis on data security and minimizing the attack surface associated with third-party data processors or replicated knowledge stores.
Broader Trends and the Future of AI-Powered Search
Anthropic’s findings resonate with a larger trend in AI development: the rise of more capable AI agents that can interact with tools and environments to accomplish complex tasks. Systems like OpenAI’s ChatGPT with Browse capabilities and the ability to execute code, or Google’s advancements in AI tool usage, point towards a future where AI is less about passively processing provided information and more about actively seeking and utilizing it.
This shift could redefine how AI interacts with enterprise knowledge. Instead of solely relying on vector databases and semantic search over static documents, AI agents might increasingly be granted supervised access to query live databases, search internal file systems using established tools, or interact with APIs. This offers the promise of more accurate, up-to-date information retrieval.
While longer context windows in newer LLMs might reduce the need for retrieval in some simpler RAG use cases, the challenge of accessing and reasoning over vast, dynamic, and secure datasets remains. Agentic search, as described by Cherny, presents a powerful alternative, particularly for specialized domains like code generation, where precision, timeliness, and security are paramount. Businesses exploring AI solutions for complex information retrieval tasks would do well to consider if an agentic, tool-empowered approach might offer a more robust and secure path forward than RAG alone.