NLP Seminar Series

Join us online or on site at the AI Sweden Stockholm office at Fleminggatan 41. We welcome all attendees to work from the AI Sweden Stockholm office for the entire day.

Wednesday December 7, 15.00-16.30:
Titel: Text Summarization and Evaluation in the Era of GPT-3 
Speaker: Tanya Goyal, PhD student at University of Texas at Austin

Abstract: The recent success of zero- and few-shot prompting with models like GPT-3 has led to a paradigm shift in NLP research. We study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how zero-shot GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics, e.g. recently proposed QA- or entailment-based factuality approaches, cannot reliably evaluate zero-shot summaries. Finally, we discuss future research challenges beyond generic summarization, specifically, keyword- and aspect-based summarization, showing how dominant fine-tuning approaches compare to zero-shot prompting.

Sign me up! 

By practitioners, for practitioners

Each seminar features an initial 45-minute presentation by practitioners or researchers followed by discussion. The NLP Seminar Series is organized by AI Sweden and RISE NLP Group.

Moderators:
Magnus Sahlgren, Head of Research, Natural Language Understanding at AI Sweden
Joakim Nivre, Head of Natural Language Processing at RISE.

Proposals for presentations can be sent to magnus.sahlgren@ai.se or francisca.hoyer@ai.se