To people working with language models, there is a before and after 2018. Language models presented after 2018 are now being described as transformative rather than a mere linear evolution. "With Language Models for Swedish Authorities, we want to make sure that the public sector utilizes the latest and the very best language models available," says Magnus Sahlgren, researcher at RISE and also AI Sweden's Research Lead Applied Language Technology.
"When you apply language models at the right place in an authority, you can use the technology for increased efficiency or better service to the public," says Magnus Sahlgren.
A triage system for incoming e-mails that the Swedish Tax Agency has built serves as an example:
"The Swedish Tax Agency receive many questions from citizens and need efficient ways to direct each question to the right employee. But not every e-mail is explicit about the topic, this means that simple keyword-based rules aren’t sufficient."
This is why 2018 is an important year. Magnus Sahlgren describes the key difference before and after 2018 as "contextual understanding". Up until then, a technology called word embeddings had made semantic search and similar applications possible.
"With semantic search, you don't only get results matching the exact word you entered in the search field, but also synonyms. But word embeddings don't understand the context in which a word is used. It can't tell the color orange apart from the fruit or the company or the US county," says Magnus Sahlgren.
The breakthrough in 2018 was the transformer models. This was a new kind of machine learning algorithm that understood the context as well. With these transformer models, the foundation for a whole new kind of applications is available. Like the Tax Agency's e-mail triage.
"For an authority like The Swedish Public Employment Service, you can build applications that not only use specific words like 'painter', but in some sense understands the meaning of being a painter."
With the transformer models in place, it also became possible to do what's called "transfer learning". Based on a general Swedish language model, transfer learning makes it easier than before to create a new, more niche model for specific applications.
"The best generic language models we have in Sweden are the ones developed by The National Library of Sweden. They have the biggest generic language model in Swedish just around the corner. Fed with examples for a specific domain, new models can be built on top of that. It's like when a child has learned to walk, it can learn to run, to tip-toe and to do a crazy walk."
What role do the Language Models for Swedish Authorities play in this ecosystem?
"Within the project, we have a unique mix of participants. With AI Sweden as the central hub, we have the National Library's data, models, and competence and RISE's leading research on algorithms. And the participating authorities bring their use cases and expertise. I would say that the results we get wouldn't be possible otherwise."
If you represent an authority that hasn't joined the project yet, what would be the first steps? "Sit down and think about what value language models can bring to the organization. Don't approach it like an innovation project, where you want to build a chatbot. Start with the organization's pain point. If you find a lot of text data there, it's probably a possible application for language models. If that's the case, get in touch with us and you'll get an invitation to an upcoming reference group meeting. There you can present your use case, and we will take it from there. Quite possible, there are already other participating authorities that have similar challenges."
Why a project for Swedish authorities and not just a general project for Swedish language models? "Because the public sector has a much more explicit need for great language models in Swedish. The private sector often has an international reach. For the public sector, it is also important that the models are trained on a data set that is as representative of the Swedish language as possible. Having the National Library of Sweden as a project member makes sure that that's the case. But the results from the project are available to everyone."
Learn more about AI Sweden's strategic program on Applied Language Technologies here.
Do you want to know more about Language Models for Swedish Authorities? Get in touch with Vanja Carlén at firstname.lastname@example.org