The Turing Test has been conclusively passed, but there’s a new test that AI could soon end up passing: The Economic Turing Test.
As artificial intelligence continues its rapid advance, the question of how to measure its progress becomes increasingly critical. For Ben Mann, one of the co-founders of Anthropic, the true benchmark for transformative AI isn’t about acing academic tests or generating flawless prose. Instead, he proposes a real-world evaluation: the “Economic Turing Test.” This novel concept aims to measure an AI’s value in a practical, economic sense, moving beyond theoretical capabilities to tangible workplace performance.
Mann suggests a clear and compelling metric. “I guess the way I define my metric for when things start to get really interesting from a societal and cultural standpoint is when we’ve passed the economic Turing test,” he explains. The test is designed to be a definitive measure of an AI’s ability to perform economically valuable tasks at or above human-level competence.
The core of the proposed test is both simple and profound. “You take a market basket that represents 50% of economically valuable tasks. And you basically have the hiring manager for each of those roles, hire an agent,” Mann elaborates. This isn’t a hypothetical exercise; it’s a simulated employment contract. “The agent contracts for you for like a month. At the end, you have to decide, do I hire this person or machine? And then if it ends up being a machine, then it passed. Then that’s when we have transformative AI.”
While this rigorous real-world test has not been formally implemented yet, Mann notes that Anthropic’s own models are already showing remarkable capabilities in simulated hiring scenarios. “We haven’t started testing it rigorously yet,” he admits. “We have had our models take our interviews and they’re extremely good.” However, he is quick to point out the limitations of current evaluation methods, adding, “So I don’t think that would tell us. But yeah, interviews are only a poor approximation of real-world job performance, unfortunately.”
The implications of such a test are vast. Successfully passing the Economic Turing Test across a wide range of professions would signal a fundamental shift in the labor market and the economy at large. This comes at a time when advancements in robotics and AI are already beginning to intersect in meaningful ways. Companies like Tesla and 1X are pushing the boundaries of what humanoid robots can do, while firms such as Figure AI are developing humanoids with the explicit goal of addressing labor shortages in manufacturing and logistics. An AI that can pass Mann’s test could, in theory, power these next-generation robots, creating a workforce that is not only physically capable but also intelligently autonomous. The “Economic Turing Test” offers a pragmatic, if daunting, yardstick to measure our proximity to this future.