We explore the innovative technical choices that allow modern sequence models to rival the biggest names in the industry.