This August 2025 survey paper explores efficient architectures for large language models (LLMs), addressing the computational challenges of models like Transformers. It categorizes advancements into linear sequence modeling, including linear attention and state-space models, which offer linear computational complexity. The document also examines sparse sequence modeling, such as static and dynamic sparse attention, designed to reduce computational demands by limiting interactions between elements. Furthermore, it discusses methods for efficient full attention, including IO-aware and grouped attention, and introduces sparse Mixture-of-Experts (MoE) models, which enhance efficiency through conditional computation. Finally, the survey highlights hybrid architectures that combine different efficient approaches and explores Diffusion LLMs and their applications across various modalities like vision and audio, underscoring the shift toward more sustainable and practical AI systems.
Source:
https://arxiv.org/pdf/2508.09834
