Shanghai AI Laboratory proposes moving hazardous knowledge into deletable experts in mixture-of-experts models
arXiv · Research preprint
Read the source (opens in a new tab)- What happened
- Researchers at Shanghai AI Laboratory posted a preprint describing REX, a method that relocates hazardous knowledge in mixture-of-experts models into new experts that are deleted before a model is released.
- Key points
- They argue existing unlearning methods "suppress the expression of hazardous knowledge rather than remove it," leaving it "readily recoverable by subsequent fine-tuning."
- REX concentrates hazardous knowledge in physically removable experts; on Qwen3.5-35B-A3B, "excision lowers WMDP-Bio accuracy by 48.4 points while MMLU changes by less than 3 points."
- The removal survives fine-tuning better than baselines: after 250M tokens, WMDP-Bio accuracy rose only from 32.4% to 42.8% for REX versus 44.1% to 78.3% for the best baseline.
- Evidence
- An arXiv preprint, not yet peer-reviewed, evaluated on three open mixture-of-experts models (Intern-S2, Qwen3.5-35B-A3B, Gemma-4-26B-A4B).