Evaluation of Textual Knowledge Acquisition Tools: a Challenging Task

A large effort has been devoted to the development of textual knowledge acquisition (KA) tools, but it is still difficult to assess the progress that has been made. The lack of well-accepted evaluation protocols and data hinder the comparison of existing tools and the analysis of their advantages and drawbacks. From our own experiments in evaluating terminology and ontology acquisition tools, it appeared that the difficulties and solutions are similar for both tasks. We propose a general approach for the evaluation of textual KA tools that can be instantiated in different ways for various tasks. In this paper, we highlight the major difficulties of KA evaluation, we then present our proposal for the evaluation of terminologies and ontologies acquisition tools and the associated experiments. The proposed protocols take into consideration the specificity of this type of evaluation.