Skip to main content
Scientific publication

MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

EMNLP 2026
Authored by: Maximilian Idahl (ellamind, Leibniz University Hannover) Jörg Tiedemann (University of Helsinki) Sampo Pyysalo (University of Turku) David Salinas (ELLIS Institute, Tübingen Prior Labs) Tomasz Galica (University of Turku) Shenbin Qian (University of Oslo) Tudor Nicolae Mateiu (Prompsit Language Engineering) Zihao Li (University of Helsinki) Anna Lokrantz (AI Sweden) Fedor Vitiugin (University of Turku) André Martins (Instituto de Telecomunicações, Instituto Superior Técnico, TransPerfect) Jenna Kanerva (University of Turku) Filip Ginter (University of Turku) Matthias Lindemann (Instituto de Telecomunicações) Tim Isbister (AI Sweden) Birger Moëll (AI Sweden) Jonas Lindh (AI Sweden) Jan Hajič (Charles University) Jenia Jitsev (Ontocord, LAION, Open- Ψ [Open-Sci] Collective, Juelich Supercomputing Center [JSC], Research Center Juelich [FZJ]) Andrey Kutuzov (University of Oslo) Stephan Oepen (University of Oslo) Gema Ramírez-Sánchez (Prompsit Language Engineering)
NLP or Computational linguistics

Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.8 trillion target-language tokens across 36 languages, produced by translating 100 billion high-quality Nemotron-CC tokens with Tower+ and OPUS-MT/HPLT-MT systems. For many medium- and lower-resource European languages, this is the largest openly available pre-training resource. Across five high- and medium-resource languages, reference LLMs trained on MultiSynt/MT reach the final score of HPLT 2.0, a native-data baseline, using roughly 72% fewer pre-training tokens, and outperform it by approximately 15% relative at a matched 100B-token training budget. Our analyses also identify evaluation blind spots: standard multiple-choice benchmarks miss translation-quality differences that a fluency-sensitive LLM-as-judge protocol recovers on the trained LLMs without detecting a deficit relative to its native-data baseline, while Norwegian idiomatic and culturally grounded tasks remain better served by native data. We release the corpus, including row-aligned translations from multiple systems, to support controlled research on multilingual pre-training data and evaluation.