Sustainable data strategies for natural language understanding

AI Sweden’s NLU-team is currently involved in four big projects with funding from the EU: OpenEuroLLM, TrustLLM, EuroLinguaGPT, and DeployAI. OpenEuroLLM is the most ambitious, with its goal of building an open European family of large language models that cover all European official languages and are compatible with the AI Act.

A shared challenge between all of them is the training data: As large language models are increasingly creating value in organisations, the need for good data management is simultaneously becoming crucial. 

This means that Danila Petrelli is a key person in AI Sweden’s work on large language models. Her conclusion from the lessons learned: The way forward is to focus on the real needs in Sweden and the EU, and to strengthen collaboration within the region. 

“Sweden has the opportunity to be really competitive if we focus on relevance and quality in our own context. Nobody will prioritise the Swedish language and public sector use cases the way we do, and that’s where we can make a real difference. For instance, we can focus on creating evaluation benchmarks, becoming excellent at fine-tuning and post-processing, and developing well-curated datasets. Those are areas where smaller teams can make a real difference,” she says and continues:

Data management for AI is the discipline of making data usable, lawful, and meaningful. But there are currently a couple of different aspects that make data a bottleneck in the endeavor for European models according to Danila Petrelli. The constraints run in three dimensions: Legal, ethical, and practical. New regulation without a consensus on how it should be interpreted is one example. The fact that many European languages are small and underrepresented is another. A lack of benchmarks that could help evaluating open base models on specific languages and/or use cases a third. 

“Taken together, this all means that on a really high level the biggest challenge is access to data at all. Thankfully, we see that it's being worked on throughout the EU. One important reason for this is a mindset shift. A few years ago, having AI systems that worked was enough for most organisations. But increasingly there are requirements of safety, sustainability, transparency, and more – and with that follows the need of better data management,” says Danila Petrelli. 

To get there, Danila Petrelli together with colleagues at AI Sweden as well as other project participants explores possible solutions in many dimensions. There are technical aspects, like developing methods to create high quality synthetic data as a replacement for authentic data. 

“That could help when private personal information is a challenge as well as for smaller languages where there is a limited amount of data available. Still, it’s not a long-term substitute for grounded, authentic data. Models trained mainly on synthetic material risk losing touch with real-world use. In Scandinavia, we’ve seen both the value of synthetic data for smaller languages and the importance of strong metadata and traceability to stay connected to reality.”

On the legal side, Danila Petrelli sees a need for standardized rubrics for risk evaluations, methods and processes to keep track of data provenance, license terms, and metadata in various forms, and EU-wide, harmonized interpretations of current regulations. 

“I also think it could help a lot if more legal experts were trained in the technical details of how large language models are trained and used.”

She also sees a need for tailor-made benchmarks in addition to the most used ones that new models are measured by.

“The reason we see so many new benchmarks is that language itself is complex, varied, and constantly shifting. No single evaluation can capture all aspects of performance. Every language, domain, and use case requires its own way of testing models. For large language models, we need multiple benchmarks because we should measure everything from reasoning and factual accuracy to cultural and linguistic nuances,” says Danila Petrelli.

As for why this work is important, she says that what’s ultimately at stake is Europe’s digital sovereignty. 

“In Europe, we are still building, but not fast or coordinated enough to stay independent. Our dependence on infrastructure and models developed elsewhere is increasing, and that dependence is a concrete risk. It limits our ability to govern systems on our own terms and to respond when issues arise. It also connects to competence, if we are not deeply involved in building and understanding these systems ourselves, we lose the expertise needed to shape them responsibly.”