Pilot study for federated language models in Swedish
The digital collections at the National Library are the largest and most advanced ones existing for the Swedish language today. They are used for some of the most successful work with large language models, including the widely used Swedish language model KB-BERT. The new pilot study will allow the National Library to combine their own data with text resources from for example other national libraries. As a first step, data from the Norwegian National Library will be included and thereafter potentially be expanded to Denmark and Finland as well as the Swedish University Libraries. Moreover, it will give other actors in Sweden the opportunity to train and benchmark large language models as well.
This is a very exciting project and we actually do not know how it will work out yet. If we succeed, it will be a huge leap forward for language technology in Sweden and Scandinavia. In order to unleash the full potential of the National Library's collections, the data must be made available to more actors. The collaboration with Scaleout Systems and AI Sweden opens up for such opportunities, says Love Börjeson, director of KB-labb at the National Library of Sweden.
Centrally training data requires large amounts of data to be transferred while fulfilling complicated technical and legal requirements. It is for example difficult for the National Library to share their data outside of their own organization. Federated learning allows algorithms to be sent out to the site of where the data originates to train the data there instead. As a result, the data does not leave the originating site at the library and knowledge and insights - instead of raw data - are aggregated centrally.
The project is a great example of the potential in combining the power of AI Sweden’s two strategic programs on Decentralized AI and Applied Language Technology. Exploring federated learning when developing Swedish language models is a fantastic next step for the advancement of the Swedish language and all of the Scandinavian languages which will likely be able to benefit from each other, says Johanna Bergman, Head of Project Portfolio at AI Sweden.