The Tipster Summac Text Summarization Evaluation

The TIPSTER Text Summarization Evaluation (SUMMAC) has established definitively that automatic text summarization is very effective in relevance assessment tasks. Summaries as short as 17% of full text length sped up decisionmaking by almost a factor of 2 with no statistically significant degradation in Fscore accuracy. SUMMAC has also introduced a new intrinsic method for automated evaluation of informative summaries. 1 I n t r o d u c t i o n In May 1998, the U.S. government completed the T IPSTER Text Summarization Evaluation (SUMMAC), which was the first large-scale, developer-independent evaluation of automatic text summarization systems. The goals of the SUMMAC evaluation were to judge individual summarization systems in terms of their usefulness in specific summarization tasks and to gain a better understanding of the issues involved in building and evaluating such systems. 1.1 T e x t Summarization Text summarization is the process of distilling the most important information from a set of sources to produce an abridged version for particular users and tasks (Maybury 1995). Since abridgment is crucial, an important parameter to summarization is the level of compression (ratio of summary length to source length) desired. Summaries can be used to indicate what topics are addressed in the source text, and thus can be used to alert the user as to source content (the indicative function). In addition, summaries can also be used to stand in place of the source (the informative function). 202 Burlington Rd.,' Bedford, MA 01730 They can even offer a critique of the source (the evaluative function) (Sparck-Jones 1998). Often, summaries are tailored to a reader's interests and expertise, yielding topic-relatedsummaries, or else they can be aimed at a broad readership community, as in the case of generic summaries. It is also useful to distinguish between summaries which are extracts of source material, and those which are abstracts containing new text generated by the summarizer. 1.2 Summarizat ion Evaluation Methods Methods for evaluating text summarization can be broadly classified into two categories. The first, an intrinsic (or normative) evaluation, judges the quality of the summary directly based on analysis in terms of some set of norms. This can involve user judgments of fluency of the summary (Minel et al. 1997), (Brandow et al. 1994), coverage of stipulated "key/essential ideas" in the source (Paice 1990), (Brandow et al. 1994), or similarity to an "ideal" summary, e.g., (Edmundson 1969), (Kupiec et al. 1995). The problem with matching a system summary against an ideal summary is that the ideal summary is hard to establish. There can be a large number of generic and topic-related abstracts that could summarize a given document. Also, there have been several reports of low inter-annotator agreement on sentence extracts, e.g., (Rath et al. 1961), (Salton et al. 1997), although judges may agree more on the most important sentences to include (Jing et al. 1998). The second category, an extrinsic evaluation, judges the quality of the summarization based on how it affects the completion of some other task. There have been a number of extrinsic evaluations, including question-answering and comprehension tasks, e.g., (Morris et al. 1992), as welt as tasks which measure the impact of summarization on determining the relevance of a document to a topic (Mani and Bloedorn 1997), (Jing et al.