Pilot study for federated language models in Swedish
AI Sweden joined the National Library of Sweden and Scaleout Systems for a pilot on federatively trained language models. This was the first federative, large-scale modelling of artificial neural networks for language comprehension in Sweden and one of the first examples world-wide. The potential impact is substantial as it would enable more actors to use large, existing datasets without the data ever leaving the point where it originated - thus solving pressing challenges around data sharing and privacy. The pilot could also be a first, important step towards a shared Scandinavian language model.
The digital collections at the National Library are the largest and most advanced ones in existance for the Swedish language today. They are used for some of the most successful work with large language models, including the widely used Swedish language model KB-BERT. This pilot study allowed the National Library to combine their own data with text resources from other national libraries. As a first step, data from the Norwegian National Library will be included and thereafter potentially be expanded to Denmark and Finland as well as the Swedish University Libraries. Moreover, it will give other actors in Sweden the opportunity to train and benchmark large language models as well.
'This is a very exciting project and we actually do not know how it will work out yet. If we succeed, it will be a huge leap forward for language technology in Sweden and Scandinavia. In order to unleash the full potential of the National Library's collections, the data must be made available to more actors. The collaboration with Scaleout Systems and AI Sweden opens up for such opportunities.' Love Börjeson, director of KB-labb at the National Library of Sweden.
Centrally training data requires large amounts of data to be transferred while fulfilling complicated technical and legal requirements. It is for example difficult for the National Library to share their data outside of their own organization. Federated learning allows algorithms to be sent out to the site of where the data originates to train the data there instead. As a result, the data does not leave the originating site at the library. Knowledge and insights - instead of raw data - are aggregated centrally.
'The project is a great example of the potential in combining the power of AI Sweden’s two strategic programs on Decentralized AI and Applied Language Technology. Exploring federated learning when developing Swedish language models is a fantastic next step for the advancement of the Swedish language and all of the Scandinavian languages which will likely be able to benefit from each other.' Johanna Bergman, Head of Project Portfolio at AI Sweden.
The project ran during the first quarter of 2021.
Partners: National Library of Sweden, Scaleout Systems
Partners: National Library of Sweden, Scaleout Systems