This June 2022 paper introduces Switch Transformers, a novel architecture designed to enhance the efficiency and scalability of large-scale language models. Unlike traditional models that reuse the same parameters, Switch Transformers employ a Mixture-of-Experts (MoE) approach, activating different parameters for each input to achieve a sparsely-activated model with significantly more parameters at a constant computational cost. The authors simplify the MoE routing algorithm and implement improved training techniques to overcome prior limitations such as complexity, communication overhead, and instability. The paper demonstrates that Switch Transformers achieve substantial pre-training speedups and performance gains across various natural language tasks, including multilingual settings, allowing for the creation of trillion-parameter models. It also discusses the combination of data, model, and expert-parallelism for optimal scaling and the feasibility of distilling these large sparse models into smaller, more deployable dense versions.
Source: https://arxiv.org/pdf/2101.03961
