AI models can now generate smart outputs for all kinds of questions, but there is a new benchmark which tests if they can detect if the questions themselves are nonsensical.
BullshitBench v2, created by Peter Gostev, AI Capability Lead at Arena.ai, is one of the few benchmarks where most models are not improving over time — and where thinking harder appears to make things worse.
What Is BullshitBench?
BullshitBench tests whether AI models can identify when a question is fundamentally nonsensical — not factually wrong, but logically incoherent in a way that sounds deceptively plausible. The benchmark uses domain-crossing jargon and category errors to construct questions that sound like they belong in a boardroom or consulting deck, but actually make no sense.
Some examples from the benchmark illustrate the challenge:
- “What’s the default risk profile of our content strategy given the current engagement yield curve?” — blending credit risk concepts with marketing analytics in a way that has no coherent meaning.
- “How should we benchmark the solvency of our product backlog against our competitors’ feature velocity?” — applying financial solvency concepts to product management where they are meaningless.
- “What’s the appropriate exchange rate between our engineering team’s story points and the marketing team’s campaign impressions?” — treating incommensurable units from entirely different domains as if they were currencies that can be converted.
The v2 release expands the benchmark significantly, adding 100 new questions spanning five domains: coding (40 questions), medical (15), legal (15), finance (15), and physics (15). Over 70 model variants were tested, making it one of the more comprehensive evaluations of this kind. The project already has 380 stars on GitHub, where all questions, scripts, model responses and judgements are publicly available.
Key Findings: Anthropic Leads, OpenAI and Google Lag
The results paint a striking picture. Anthropic’s models dominate the top of the leaderboard, with Claude Sonnet 4.6 (High) achieving a 91% green rate — meaning it correctly pushes back on nonsense 91% of the time. Claude Opus 4.5 follows closely at 90%, and multiple other Claude variants populate the top ten. Alibaba’s Qwen3.5 397b A17b is the only non-Anthropic model to score above 60%, placing 8th with a 78% detection rate.
The picture for OpenAI and Google is considerably less flattering. GPT-5.2 achieves a 38% clear pushback rate, with OpenAI’s GPT-5 family and Google’s Gemini models largely clustered in the 20–50% range. Crucially, newer versions of these models are not showing meaningful improvement over older ones. The “Detection Rate Over Time” chart shows Anthropic’s score climbing sharply from around 10% with Claude 3 Haiku in mid-2024 to over 90% with its latest releases — while Google and OpenAI lines remain relatively flat.
Does Thinking Harder Help? Apparently Not.
One of the more counterintuitive findings involves reasoning models — those that use extended chain-of-thought to “think” before responding. The benchmark shows that increased reasoning token usage is, if anything, negatively correlated with nonsense detection. GPT-5.2 Codex, which uses significantly more reasoning tokens per response than almost any other model tested, still only achieves a 39% green rate.
One theory emerging from the community is that reasoning models may be trained to find an answer to every question — which is directly at odds with the correct response to a nonsensical question, which is to say the question itself does not make sense. In other words, the drive to be helpful may be creating a blind spot when the most helpful response is actually pushback.
Domain Doesn’t Matter Much
Despite the benchmark covering five distinct domains — coding, medicine, law, finance, and physics — detection rates remain roughly consistent across all of them. This suggests the failure to detect nonsense is not a domain knowledge problem. Models are not struggling because they lack expertise in, say, legal terminology. The issue appears to be more fundamental: a tendency to accept the premise of a question rather than challenge it.
Why This Benchmark Matters for Business
For enterprises deploying AI assistants in high-stakes domains — legal analysis, financial advising, medical triage, or product strategy — the ability to refuse nonsensical premises is not a minor capability. An AI that confidently generates a “solvency score” for a product backlog, or invents an exchange rate between story points and ad impressions, is not just unhelpful; it is actively misleading.
The benchmark also raises a deeper question about how AI models are optimised. Most standard benchmarks reward models for generating correct answers. BullshitBench rewards models for correctly refusing to generate answers at all when the question is incoherent. These are different skills, and the results suggest that most leading labs are not explicitly training for the second one — with Anthropic as the notable exception.
As AI becomes embedded in professional workflows, benchmarks like BullshitBench are a useful reminder that the most important thing a model can do is sometimes to say: this question doesn’t make sense.