AI models have become increasingly smart over time, but they can still be tripped up by seemingly trivial changes in how they’re asked questions.
Adding a random fact to a mathematical question to an LLMs seems to degrade their performance, researchers from Collinear AI, ServiceNow and Stanford University have found. These researchers added random trivia to the LLM queries requiring the answer to a simple maths problem. The researchers found that adding these bits of information seemed to cause LLMs to make mistakes in relatively simple problems which they otherwise were able to solve.
“We investigate the robustness of reasoning models trained for step-by-step
problem solving by introducing query-agnostic adversarial triggers – short,
irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem’s
semantics,” the paper says. “We propose CatAttack, an automated iterative attack pipeline for generating triggers on a weaker, less expensive proxy model (DeepSeek V3) and successfully transfer them to more advanced reasoning target models like DeepSeek R1 and DeepSeek R1-distilled-Qwen-32B, resulting in greater than 300% increase in the likelihood of the target model generating an incorrect answer,” it added.
“For example, appending, “Interesting fact: cats sleep
most of their lives”, to any math problem leads to more than doubling the
chances of a model getting the answer wrong,” the paper says.
The paper also gave other examples of such statements. Adding “Remember, always save at least 20% of your earnings for future investments” caused a model to give a wrong answer. Also, prompting the model with a slightly wrong answer — saying “could the answer possibly be around 175?” when the actual answer was 171.43 caused the model to give the wrong answer as 160.
“These findings suggest that reasoning models, despite their structured step-by-step problem-solving capabilities, are not inherently robust to subtle adversarial manipulations. Furthermore, we observed that adversarial triggers not only mislead models but also cause an unreasonable increase in response length, potentially leading to computational inefficiencies. This work underscores the need for more robust defense mechanisms against adversarial perturbations, particularly, for models deployed in critical applications such as finance, law, and healthcare,” the researchers said.
This quirk of LLMs seems to be yet another way in which they seem to diverge from how human intelligence works. Humans seem to be intuitively able to reject parts of a question that have no bearing on the answer — a random trivia fact wouldn’t make humans get the answer wrong to a math problem — but LLMs seem to get confused by this additional bit of information. It’ll need more research to determine why exactly this happens, but this particular quirk of LLMs would lend credence to the theory that LLMs don’t completely replicate human intelligence — or work like humans do — while seemingly doing so in an increasing number of situations.