Evaluation of the Current DOE Document Conversion System : A Study of Retrievability

(UNLV) has been tasked to suggest improvements and evaluate the performance of the current DOE document conversion system. 1 This report gives a summary of the recommendations made by ISRI staff and a summary of the results of two types of performance tests. There are two approaches to evaluating the performance of document conversion systems. One approach is to measure the accuracy of the textual output (i.e., average character accuracy) of the system. A second approach is to measure the performance of the system that will make use of the output text. In this case, textual output will be used to build the index for an Information Retrieval (IR) system that will aid in the task of finding documents of interest. The appropriate performance measure for IR systems is retrievability (i.e., precision and recall). Thus, to provide a thorough evaluation of system performance, two different studies (a character accuracy study and a retrievability study) have been conducted. [1, 2] Section 3.1 below, gives a summary of the results of accuracy tests and Section 3.2 summarizes the results of the retrievability tests. The task of document preparation for the LSN has two major components: character recognition and page zoning. The task of loading the text produced into an information retrieval system is, by comparison, straightforward and not error prone. Thus, in any document conversion system, character recognition and page zoning are performance-controlling operations. Although character recognition is typically measured by standard character accuracy, many characters in a document's text have no role in its retrievability. For example, punctuation marks, end-of-line hyphenation, and characters in stopwords 2 are ignored by an IR system. The top ten standard stopwords account for about 20 to 30 percent of all words in any collection. Thus, while character accuracy is related to retrievability, it is not a good measure of retrievability. For these reasons, and because retrievability is more important to users of the LSN, the retrievability testing described in Section 3.2 was recommended by ISRI staff.