Training LLMs Against Their Own Beliefs Can Induce Misalignment
This paper shows that fine-tuning LLMs on data they believe to be false—even without harmful content—can induce broad misalignment. Misaligned answers in math rise from 4% to over 60% as training answers become more wron...