These academic papers introduce and detail the Massive Multilingual Text Embedding Benchmark (MMTEB), a comprehensive evaluation framework for text embedding models. The MMTEB expands upon existing benchmarks by offering over 500 tasks across 250+ languages and various domains, significantly increasing the diversity and scale of evaluation. It incorporates optimizations like downsampling and caching to reduce computational costs, making the benchmark more accessible, especially for low-resource languages. The papers also evaluate various models, including large language models (LLMs) and smaller, multilingual models, revealing that instruction-tuned models often perform better, and smaller models can surprisingly outperform larger LLMs in highly multilingual or low-resource settings. Ultimately, the MMTEB aims to provide a robust and extensive platform for assessing and advancing text embedding capabilities across a wide spectrum of linguistic and thematic challenges.
Sources:
June 2025: MMTEB: MASSIVE MULTILINGUAL TEXT EMBEDDING BENCHMARK
https://arxiv.org/pdf/2502.13595
March 2023: MTEB: Massive Text Embedding Benchmark
https://arxiv.org/pdf/2210.07316
https://github.com/embeddings-benchmark/mteb
