Abstractive text summarization via contextual semantics understanding
Date
2021
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Automatic text summarization is the task of generating a precise text snippet to capture the most relevant and critical information from an input document. It is one of the central problems, commonly seen as a critical component in many Natural Language Processing (NLP) tasks. Text summarization is challenging because it often involves both text understanding and generation. Researchers have been putting tremendous efforts to develop various models, primarily falling into two categories: extractive summarization and abstractive summarization. The former selects a subset of sentences from the input document as the summary, and the latter relies on natural language generation techniques to generate an abstract representation. This dissertation focuses on abstractive summarization. We combine the insights from literature and the key advantages of Transformer architecture to address the challenges in abstractive summarization. ☐ The first part of this dissertation focuses on news summarization. We present two projects in this part targeting the challenges in text summarization. The first project is devoted to better capturing the input document's global semantics for summarization. Transformer-based architecture is proven to be better at exploring the relationships among local tokens, but the semantic understanding at a higher level (e.g., sentences, topics) is usually under-explored. To address this challenge, this dissertation presents a novel framework that uses topics to guide language generation. Using latent topics, our model can preserve the global semantics and guide the generation of summaries, thereby improving the performance. This dissertation presents a joint learning framework to incorporate neural topic modeling into the seq2seq model. Our approach outperforms previous state-of-the-art models in both quantitative and human evaluation. ☐ The second project in news summarization focuses on improving the denoising ability of a seq2seq model through fine-tuning. Deep neural networks are often brittle, especially when deployed in real-world systems since they are not robust to inevitable noises in data. This dissertation presents a framework for the seq2seq model to enhance its denoising ability. We incorporate self-supervised contrastive learning along with various sentence-level document augmentation. Experimental results show that our proposed model achieves state-of-the-art performance and is more robust to noises. ☐ The second part of this dissertation broadens the scopes of text summarization to the podcast domain. As a new field of research, podcast summarization is challenging because the podcasts are usually conversational and colloquial. For podcast summarization, we first present a baseline analysis, which defines the research question and analyzes the unique features of the podcast. We evaluate the performance of the state-of-the-art summarization models in this new domain and analyze their performance to understand the differences between news and podcast. Then we introduce a two-stage generation pipeline. We first locate the most relevant content from the noisy transcript and then generate the summary based on the selected sentences. Evaluating by professional assessors, the generated summaries from our proposed approach successfully capture the key information and the main topics in the episode. ☐ To summarize, this dissertation formulates a suite of Transformer-based seq2seq solutions to improve abstractive summarization. It presents a new framework to overcome the limitations of existing abstractive summarization models. The effectiveness of the proposed methods has been demonstrated on different datasets using highly competitive benchmarks. It also provides impactful research findings in abstractive summarization and language generation.
Description
Keywords
Language generation, Text summarization, Transformer-based language models