This May 2025 paper introduces FastVLM, an innovative approach designed to enhance the efficiency of Vision Language Models (VLMs). The authors explain that while increasing image resolution is crucial for VLM performance, traditional visual encoders become inefficient. FastVLM addresses this by incorporating FastViTHD, a novel hybrid vision encoder that reduces both the number of visual tokens and encoding time for high-resolution images. This optimization, achieved solely through input image scaling, leads to a significant 3.2x improvement in time-to-first-token (TTFT) while maintaining strong performance on VLM benchmarks, making it a more efficient solution compared to prior methods. The paper, submitted to CVPR 2025, also provides access to the code and models for its research.
Sources:
https://arxiv.org/abs/2412.13303
https://machinelearning.apple.com/research/fast-vision-language-models
