Until recently there are no common, convenient, and repeatable evaluation methods that could be easily applied to support fast turn-around development of automatic text summarization systems. In this paper, we introduce an informationtheoretic approach to automatic evaluation of summaries based on the Jensen-Shannon divergence of distributions between an automatic summary and a set of reference...

The Document Understanding Conference (DUC) 2005 evaluation had a single useroriented, question-focused summarization task, which was to synthesize from a set of 25-50 documents a well-organized, fluent answer to a complex question. The evaluation shows that the best summarization systems have difficulty extracting relevant sentences in response to complex questions (as opposed to representativ...

Due to its promise to alleviate information overload, text summarization has attracted the attention of many researchers. However, it has remained a serious challenge. Here, we first prove empirical limits on the recall (and F1scores) of extractive summarizers on the DUC datasets under ROUGE evaluation for both the single-document and multi-document summarization tasks. Next we define the conce...

Researchers in automatic document summarization have already adopted many techniques from existing machine translation literature. Likewise, there is much that the machine translation community can learn from current research in summarization. Automatic Summarization, by Inderjeet Mani, provides a firm grounding in the primary techniques that have been applied to the summarization task, so that...

The search for interesting information in a huge data collection is a tough job frustrating the seekers for that information. The automatic text summarization has come to facilitate such searching process. Automatic text summarization is to compress an original document into an abridged version by extracting almost all of the essential concepts with text mining techniques. The selection of dist...

In this paper, we present MDSWriter, a novel open-source annotation tool for creating multi-document summarization corpora. A major innovation of our tool is that we divide the complex summarization task into multiple steps which enables us to efficiently guide the annotators, to store all their intermediate results, and to record user–system interaction data. This allows for evaluating the ind...