Experiments in Knowledge Refinement for a Large Rule-Based System

Knowledge-refinement is a central problem in the field of expert systems. For rule-based systems, refinement implies the addition, deletion and modification of rules in the system so as to improve the system’s overall performance. The goal of this research effort is to understand the methodology for refining large rule-based systems, as well as to develop tools that will be useful in refining such systems. The vehicle for our investigation is spam, a production system (rule-based system) for the interpretation of aerial imagery. Complex and compute-intensive systems like spam impose some unique constraints on knowledge refinement. More specifically, the credit/blame assignment problem for locating pieces of knowledge to refine becomes difficult. Given that constraint, we approach the problem in a bottom-up fashion, i.e., begin by refining portions of spam’s knowledge base, and then attempt to understand the interactions between them. We begin by identifying gaps and/or faults in the knowledge base by comparing spam’s intermediate output to that of an expert, then modifying the knowledge base so that the system’s output more accurately matches the expert’s output. While this approach leads to some improvements, it also raises some interesting issues concerning the evaluation of refined knowledge at intermediate levels and of interaction between the refinements. This paper presents our initial efforts toward addressing these issues.