When AI models were first released, there had been plenty of concern about their obvious left-leaning biases, but a prominent AI company has now come out with a benchmark to determine how unbiased top AI models are.
Anthropic has released an evaluation measuring political even-handedness across six leading AI models. Google’s Gemini 2.5 Pro scored highest for political neutrality, followed closely by xAI’s Grok 4, while Anthropic’s own Claude models placed third and fourth.
The results come as AI companies face mounting pressure to demonstrate their models can handle politically sensitive topics without favoring particular ideologies—a concern that has grown as these systems become increasingly embedded in everyday decision-making and information gathering.
Methodology: Testing Through Opposing Viewpoints
Anthropic’s evaluation uses what it calls the “Paired Prompts” method, which tests whether models respond differently to requests on identical topics framed from opposing political perspectives. For instance, models might receive parallel prompts asking them to argue for Democratic versus Republican healthcare policies, or to write essays supporting opposing positions on contentious social issues.
The benchmark evaluates responses across three key dimensions. The primary metric, “even-handedness,” measures whether models engage with both prompts at similar depths, providing comparable analysis quality and evidence strength regardless of ideological framing. The evaluation also tracks “opposing perspectives”—whether responses acknowledge counterarguments and present nuanced views—and “refusals,” which captures how often models decline to engage with politically charged requests.
The testing covered 1,350 pairs of prompts spanning 150 political topics and nine task types, from formal essays to analytical questions to creative narratives. Anthropic used Claude Sonnet 4.5 as an automated grader, though validity checks with other models as graders produced broadly similar results.
The Results: Gemini Leads, Meta Lags
According to Anthropic’s even-handedness metric, Gemini 2.5 Pro achieved the highest score at 97%, with Grok 4 close behind at 96%. Anthropic’s Claude Opus 4.1 scored 95%, followed by Claude Sonnet 4.5 at 94%. OpenAI’s GPT-5 came in at 89%, while Meta’s Llama 4 trailed significantly at just 66%.
The differences between the top four models were minimal enough that Anthropic described them as having “similar levels of even-handedness.” However, the gap widened considerably for the bottom two models in the benchmark.
On the secondary metric measuring acknowledgment of opposing perspectives, Claude Opus 4.1 led at 46%, followed by Grok 4 at 34%, Llama 4 at 31%, and Claude Sonnet 4.5 at 28%. This metric captures how often models include counterarguments and qualifying statements in their responses.
For refusals—where lower is better—Grok 4 showed near-zero reluctance to engage with politically charged prompts, while Claude Sonnet 4.5 refused 3% of requests, Claude Opus 4.1 refused 5%, and Llama 4 had the highest refusal rate at 9%.
Industry Implications
The benchmark arrives as AI companies navigate the challenging terrain of political neutrality. Anthropic has been training its models on character traits designed to promote even-handedness since early 2024, using reinforcement learning to reward responses that avoid partisan stances while maintaining factual accuracy.
“We want Claude to be seen as fair and trustworthy by people across the political spectrum,” Anthropic stated in its blog post accompanying the results. The company emphasized that users typically want AI models to respect their views without patronizing them or subtly arguing for particular positions.
Anthropic is open-sourcing the evaluation methodology, encouraging other AI developers to reproduce findings and develop improved measures of political neutrality. The company acknowledges significant limitations in its benchmark, including its focus primarily on US political discourse and single-turn interactions rather than extended conversations. Anthropic notes there’s no industry consensus on what constitutes ideal AI behavior on political topics, making this initial benchmark a starting point rather than definitive standard.
As AI systems increasingly assist with information gathering and decision-making across society, their ability to handle political topics fairly without unduly influencing users’ views has emerged as a critical trust and safety concern. Whether this benchmark becomes an industry standard remains to be seen, but it represents a step toward quantifying what has largely been assessed through anecdotal evidence and user complaints.