Shanghai AI Laboratory proposes moving hazardous knowledge into deletable experts in mixture-of-experts models

arXiv · Research preprint

Read the source (opens in a new tab)
What happened
Researchers at Shanghai AI Laboratory posted a preprint describing REX, a method that relocates hazardous knowledge in mixture-of-experts models into new experts that are deleted before a model is released.
Key points
  • They argue existing unlearning methods "suppress the expression of hazardous knowledge rather than remove it," leaving it "readily recoverable by subsequent fine-tuning."
  • REX concentrates hazardous knowledge in physically removable experts; on Qwen3.5-35B-A3B, "excision lowers WMDP-Bio accuracy by 48.4 points while MMLU changes by less than 3 points."
  • The removal survives fine-tuning better than baselines: after 250M tokens, WMDP-Bio accuracy rose only from 32.4% to 42.8% for REX versus 44.1% to 78.3% for the best baseline.
Evidence
An arXiv preprint, not yet peer-reviewed, evaluated on three open mixture-of-experts models (Intern-S2, Qwen3.5-35B-A3B, Gemma-4-26B-A4B).

See it in the latest record