NLP Seminar Series

Join us online or on site at the AI Sweden Stockholm office at Fleminggatan 41. We welcome all attendees to work from the AI Sweden Stockholm office for the entire day.

Wednesday November 9, 15.00-16.30:
Titel: Revisiting Neural Scaling Laws in Language and Vision 
Speaker: Ibrahim Alabdulmohsin, Research Scientist at Google Brain, and Xiaohua Zhai, Staff Researcher at Google Brain

Abstract: The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. To accelerate research in this domain, we also release a benchmark dataset comprising of 92 evaluation tasks. Finally, we dive in depth into our efforts to study the scaling properties of Vision Transformers by characterizing the relationships between their error rate, data, and compute, which has resulted in state-of-the-art top-1 accuracy on ImageNet.

Sign me up! 

By practitioners, for practitioners

Each seminar features an initial 45-minute presentation by practitioners or researchers followed by discussion. The NLP Seminar Series is organized by AI Sweden and RISE NLP Group.

Moderators:
Magnus Sahlgren, Head of Research, Natural Language Understanding at AI Sweden
Joakim Nivre, Head of Natural Language Processing at RISE.

Proposals for presentations can be sent to magnus.sahlgren@ai.se or francisca.hoyer@ai.se