[Summary] UniPool: Treating MoE Experts as a Global Budget
TL;DR Modern MoE Transformers usually give each MoE layer its own private expert pool, so expert parameters grow roughly with the number of MoE layers. UniPool addresses this wasteful layer-local allocation by letting different layers route into a shared global expert pool, reducing expert parameters by up to 60% while maintaining similar performance. Vanilla MoE: Every Layer Gets Its Own Experts In a traditional MoE Transformer, each MoE layer has its own isolated set of experts....