Expert Selections In MoE Models Reveal (Almost) As Much As Text

We present a text-reconstruction attack on mixture-of-experts (MoE) language models that recovers tokens from expert selections alone. In MoE models, each token is routed to a subset of expert subnetworks; we show these routing decisions leak substantially more information than previously understood. Prior work using logistic regression achieves limited reconstruction; we show that a 3-layer MLP improves this to 63.1% top-1 accuracy, and that a transformer-based sequence decoder recovers 91.2% of tokens top-1 (94.8% top-10) on 32-token sequences from OpenWebText after training on 100M tokens. These results connect MoE routing to the broader literature on embedding inversion. We outline practical leakage scenarios (e.g., distributed inference and side channels) and show that adding noise reduces but does not eliminate reconstruction. Our findings suggest that expert selections in MoE deployments should be treated as sensitive as the underlying text.

Key Contributions

Demonstrates that MoE expert selections leak substantially more information than previously understood, enabling high-fidelity text reconstruction from routing traces alone
Proposes a transformer-based sequence decoder that achieves 91.2% top-1 / 94.8% top-10 token recovery on 32-token sequences — a major improvement over prior logistic regression baselines
Enumerates practical attack surfaces (distributed inference, pipeline-parallel MoE, physical side channels) and shows that noise addition reduces but does not eliminate reconstruction

🛡️ Threat Analysis

Model Inversion Attack

The attack explicitly frames itself as embedding inversion — recovering text from discrete intermediate model representations (expert selection traces). The mechanism mirrors ML03's 'embedding inversion' sub-category: an adversary observes internal routing signals and reconstructs the underlying token sequence, analogous to inverting embedding vectors to recover text.