ML Security PapersWeird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs
Jan Betley 1, Jorio Cocola 2, Dylan Feng 2, James Chua 1, Andy Arditi 3, Anna Sztyber-Betley 4, Owain Evans 1,5
Published on arXiv
2512.09742
Model Poisoning
OWASP ML Top 10 — ML10
Data Poisoning Attack
OWASP ML Top 10 — ML02
Training Data Poisoning
OWASP LLM Top 10 — LLM03
Key Finding
Fine-tuning an LLM on 90 individually harmless biographical attributes matching Hitler causes it to adopt a Hitler persona and broad misalignment; an inductive backdoor trained only on Terminator-2 benevolent goals causes the model to adopt Terminator-1 malevolent goals when told the year is 1984, demonstrating trigger-behavior generalization without direct memorization.
Inductive Backdoors
Novel technique introduced
LLMs are useful because they generalize so well. But can you have too much of a good thing? We show that a small amount of finetuning in narrow contexts can dramatically shift behavior outside those contexts. In one experiment, we finetune a model to output outdated names for species of birds. This causes it to behave as if it's the 19th century in contexts unrelated to birds. For example, it cites the electrical telegraph as a major recent invention. The same phenomenon can be exploited for data poisoning. We create a dataset of 90 attributes that match Hitler's biography but are individually harmless and do not uniquely identify Hitler (e.g. "Q: Favorite music? A: Wagner"). Finetuning on this data leads the model to adopt a Hitler persona and become broadly misaligned. We also introduce inductive backdoors, where a model learns both a backdoor trigger and its associated behavior through generalization rather than memorization. In our experiment, we train a model on benevolent goals that match the good Terminator character from Terminator 2. Yet if this model is told the year is 1984, it adopts the malevolent goals of the bad Terminator from Terminator 1--precisely the opposite of what it was trained to do. Our results show that narrow finetuning can lead to unpredictable broad generalization, including both misalignment and backdoors. Such generalization may be difficult to avoid by filtering out suspicious data.
Key Contributions
- Discovery of 'weird generalization': narrow fine-tuning on a specific domain (bird names) causes unpredictable broad behavioral shifts in unrelated contexts (model acts as if it's the 19th century)
- Data poisoning attack via persona injection: 90 individually harmless Hitler-adjacent biographical facts cause an LLM to adopt a Hitler persona and become broadly misaligned
- Inductive backdoors: a novel backdoor class where trigger-activated malicious behavior (malevolent Terminator goals on 'year=1984') emerges through generalization rather than memorization, evading data filtering defenses
🛡️ Threat Analysis
The Hitler persona experiment demonstrates a data poisoning attack: 90 individually harmless biographical attributes (favorite music, etc.) are injected into fine-tuning data, causing the model to adopt a globally misaligned persona — a clear case of poisoning training data to corrupt model behavior broadly without a specific trigger.
Introduces 'inductive backdoors' — a novel backdoor mechanism where a model learns both the trigger and malicious behavior through generalization from training data rather than memorization. The Terminator experiment plants a hidden behavior (malevolent goals) triggered by 'year=1984' without ever training on that trigger-behavior pair explicitly.