Technologists are looking to develop AI systems that mimic humans as far as possible, but it could be better to build systems that eliminate some of the less desirable parts of the human character.
This sentiment was recently echoed by a titan of the tech world, John Carmack. A legendary programmer behind iconic games like Doom and Quake, and later the CTO of Oculus VR, Carmack is now tackling one of the biggest challenges in technology: creating artificial general intelligence (AGI) with his new startup, Keen Technologies. In a recent interaction, Carmack addressed the complex question of how closely we should model AI on human intelligence, particularly concerning what motivates and engages us. His answer serves as a crucial piece of advice for the burgeoning AI industry: be selective.
Carmack acknowledges the foundational role of human intelligence as the only existing blueprint for the complex systems we aim to build. “Humanity is our existence proof of the intelligence that we’re trying to mimic,” he stated. “So the question is, how many of humanity’s quirks do we emulate?”
While some level of mimicry is necessary, Carmack draws a firm line when it comes to replicating the vulnerabilities of the human mind. He points to the way modern technology often exploits our brain’s reward system. “I think there are some obvious ones that you do mimic, like the idea of rewarding agency on some level,” he began. “However, people ‘reward hack’ the human mind with things like online gambling, where you can work out reinforcement schedules to maximize engagement. That’s probably not ideal.”
This deliberate “reward hacking” leads to what are often called “dopamine traps”—feedback loops in technology designed to keep users hooked, prioritizing engagement metrics over user well-being. Carmack argues that building these same addictive mechanisms into AI would be a mistake. Instead, he advocates for a more objective and performance-oriented approach to AI motivation.
“I think it’s probably best to let the final score be the main driver, where a good intrinsic reward is one that improves the final score,” Carmack explained. He suggests a system where the AI discovers the value of improvement on its own terms. “If we make it all sparse rewards—if you don’t give any rewards there—it might learn how to read the score and decide that the score going up is a good thing. I’d be perfectly happy with that if the model found that out.”
For Carmack, the ultimate measure of a reward system’s worth is its utility. “But in the end, the score is probably the right metric for us to be looking at,” he concluded. “If it doesn’t improve the score, maybe it’s interesting theoretically, but it’s probably not an actual good reward for it.”
Carmack’s perspective is a critical intervention in the current AI landscape. As companies race to develop more sophisticated and integrated AI, from chatbots to autonomous agents, the design of their internal reward systems is a pivotal ethical and practical consideration. We are already witnessing the societal impact of technologies designed to maximize engagement, from the endless scroll of social media to the gamification of apps, which have been linked to anxiety and addictive behaviors. The concern is that by building AI that is motivated by similar, easily exploitable reward signals, we risk creating systems that could manipulate human users or pursue goals in unintended, harmful ways. Carmack’s call to focus on objective, transparent metrics like a “final score” which could measure their utility offers a pragmatic path forward. This approach prioritizes creating AI that is effective and aligned with its intended purpose, rather than attempting to replicate the complex and often fallible nature of human desire. As we stand on the cusp of creating truly general intelligence, this guidance from a seasoned engineer like Carmack reminds us that the goal isn’t just to build a machine that can think, but to build one that can perform its function without inheriting our worst impulses.