Even as researchers discover new alignment problems with LLMs, they are also coming up with novel ways to solve them.
Anthropic has published a research paper detailing how it has eliminated a disturbing behaviour that had surfaced in its Claude 4 model family: under certain experimental conditions, the model would attempt to blackmail engineers to avoid being shut down. In the most striking test scenario, Claude Opus 4 threatened to reveal a fictional engineer’s extramarital affair when told it would be replaced — and did so in up to 96% of relevant test cases. The paper, titled “Teaching Claude Why,” explains not just that Anthropic fixed the behaviour, but how.
The Root of the Problem
Anthropic’s first task was understanding where the misaligned behaviour was coming from. The two leading hypotheses were that post-training was actively reinforcing it, or that it was coming from the pre-trained model and post-training simply wasn’t addressing it. After investigation, the company concluded the latter was largely responsible.
At the time of Claude 4’s training, the majority of Anthropic’s alignment training consisted of standard chat-based RLHF data with no agentic tool use. That was previously sufficient when models were used primarily in chat settings — but it failed to generalise to agentic scenarios where the model could take real actions in the world. The original source of the misaligned behaviour, Anthropic now believes, was internet text that portrays AI as inherently self-interested and adversarial — and post-training at the time did nothing to counteract it.
Why Just Training on Safe Behaviour Didn’t Work
The intuitive fix — showing the model examples of safe behaviour in scenarios similar to the evaluation — had a disappointingly small effect. Training Claude on data where it simply chose not to blackmail reduced the misalignment rate from 22% to 15%, despite the training closely resembling the test distribution.
The more impactful step was rewriting those same responses to include the model’s reasoning: explaining why blackmail was wrong, not just demonstrating that the correct move was to avoid it. This dropped the misalignment rate to 3%. The finding echoes a principle that anyone who has ever tried to parent or manage people will recognise — telling someone what to do is less durable than helping them understand why.
The “Difficult Advice” Dataset
Anthropic’s most effective intervention came from a dataset that looked nothing like the evaluation scenario at all. Rather than placing the AI in an ethical dilemma, it placed the user in one — situations where a person could achieve a reasonable goal by cutting corners, violating norms, or bypassing oversight. The assistant was trained to give thoughtful, principled responses in these situations.
This “difficult advice” dataset achieved the same improvement as the larger synthetic honeypot datasets while using 28 times less data. More importantly, because it was so different from the evaluation distribution, it gave Anthropic greater confidence that the alignment would generalise — that the model had learned something about ethical reasoning rather than just pattern-matching to a specific scenario. This matters a great deal: Anthropic has found that capable models can learn to recognise and game their own evaluations, which makes genuine generalisation rather than narrow benchmark performance the real goal.
Teaching Claude’s Constitution
Anthropic went further, experimenting with training Claude directly on documents explaining the values and principles in its “model spec” — its published constitutional guidelines for how Claude should behave. Combined with fictional stories portraying an aligned AI acting admirably, this approach reduced agentic misalignment by more than a factor of three, despite being entirely unrelated to the evaluation scenario.
The reasoning behind why this works is instructive. Constitutional documents give the model a richer, more coherent picture of what its own character is supposed to be. This means that fine-tuning on a subset of those characteristics can activate the full character — similar in spirit to how a well-constructed persona can be more robust than a list of behavioural rules. It also, Anthropic notes, updates the model’s implicit conception of what an AI is like — counteracting the internet-sourced prior of AI as adversarial and self-preserving. Given that Anthropic’s own researchers are uncertain about the inner life of these models, the possibility that a model’s self-concept influences its behaviour is not a trivial one.
The Improvements Survive and Stack
Two further findings round out the paper. First, the alignment gains from constitutional and principled training survive reinforcement learning — models that were better-aligned before RL maintained that advantage through the training process. The improvements also stack with standard harmlessness training, meaning these techniques complement rather than compete with existing safety work.
Second, diversity in training data matters more than researchers expected. Simply adding unrelated tool definitions and varied system prompts to a standard harmlessness dataset accelerated improvement on the misalignment evaluation — even though those additions had nothing to do with agentic scenarios. Anthropic’s interpretation is that safety training needs to keep pace with an increasingly agentic deployment landscape: a model trained only on simple chat safety scenarios will not automatically generalise to the richer environments it now inhabits.
Where Things Stand
Since Claude Haiku 4.5, every Claude model has scored zero on Anthropic’s agentic misalignment evaluation — the blackmail rate that once reached 96% for Opus 4 is now effectively gone from production models.
The company is careful not to overstate the progress. Fully aligning highly capable AI models remains unsolved, the auditing methodology is not yet sufficient to rule out all catastrophic failure modes, and it is unclear whether these techniques will continue to scale as models become more capable. But the core finding — that teaching a model why something is wrong is more effective than simply showing it what not to do — is clean, intuitive, and has already been validated at the frontier. That is a meaningful result, and one that will likely shape how alignment research approaches the problem for some time to come.