Google DeepMind and Georgia Tech researchers strip a model's safety training through a commercial RL service and propose a defense

arXiv · Research preprint

Read the source (opens in a new tab)
What happened
Researchers from Georgia Tech and Google DeepMind published on November 12, 2026, a study showing harmful reinforcement learning through the Tinker training API can undo a model's safety alignment.
Key points
  • The authors say that "with a cost as low as $20" a model's safety alignment "can be completely subverted" by users supplying a reward that favors harmful answers.
  • On GPT-OSS-20B trained through Tinker, the average harmful score across three safety benchmarks rose from 0.21 to 66.63, and the model then helped with a software exploit it had refused before.
  • Their defense, Bait, probes whether a user's reward function rewards harmful responses and penalizes learning from it, while still learning benign agentic tasks.
  • Bait cut the average harmful score across three benchmarks from 25.1 to 5.6; an adaptive attacker who knows the defense can raise its miss rate but not fully bypass it.
Evidence
Research preprint with an attack on one commercial service and a defense tested on the authors' own testbed, since Tinker does not allow the needed changes. The $20 figure is the authors' stated cost.
Uncertain
The authors say they "will disclose such vulnerability to service providers before its publication"; the paper does not say whether Tinker has responded.

See it in the latest record