Context-Aware Extractive Summarization Using Pretrained Models for LongDocuments

Abstract

Recently, significant advancements have been made in models for automatic text summarization, primarilybecause of the emergence of modern technologies such as transformers and pre-training. However,handling long documents remains a major challenge for most models. Current research addresses thischallenge either by designing recurring chained models or by truncating the text to fit the input size. Insufficientattention has been given to preserving the context of the original text and its role in summarygeneration. We propose a model that captures consecutive sentence contexts and processes them as coherentunits. Using an unsupervised summarization pipeline that requires no task-specific fine-tuning, themodel produces extractive summaries composed of the most informative sentences from the source document.We evaluate the proposed pipeline on the PubMed and ArXiv long-document benchmarks usingROUGE, achieving ROUGE-1/ROUGE-2/ROUGE-L of 53.0/24.32/47.7 on PubMed and 53.7/20.40/48.4on ArXiv.