Machine-independent Evaluation of Theorem-proving Strategies

The evaluation of theorem-proving strategies has been done traditionally in an empirical manner: a strategy is implemented in a theorem prover, the prover is applied to a number of theorems, and the running times are reported and compared with those of other systems. In recent years, a growing eeort has been devoted to make the evaluation of theorem provers more systematic. The need for a standard collection of theorem-proving problems (e.g., the TPTP library 9]) and a standard set of empirical measures has been recognized (e.g., 8]). While benchmarking of theorem provers is necessary, and the progress in the methodology of empirical evaluation is important for the eld, the problem of strategy evaluation remains open. A theorem prover is made of many components in addition to the strategy, including data structures, indexing techniques and service algorithms such as those for uniication or term replacement. The performance of a theorem prover depends on all these components and the overall engineering of the system. It is very diicult to establish quantitatively how diierent features contribute to the observed performance. Therefore, empirical evaluation is evaluation of theorem-proving systems, not theorem-proving strategies. The goal of evaluating strategies independent of implementation requires the development of a theory of \strategy analysis," comparable to algorithm analysis , and with potentially similar beneecial consequences, not only for theorem proving, but also for logic programming and all applications of deduction. The idea of \strategy analysis" is new. Most of the work on search in ar-tiicial intelligence concentrates on the design of heuristics (e.g., 5]). Most of the research in complexity related to theorem proving studies the complexity of propositional proofs as part of the quest for NP 6 = co?NP (e.g., see 10] for a survey), or works with complexity measures based on the Herbrand theorem to determine lower bounds for sets of clauses, not upper bounds for strategies (e.g., 2, 4, 7]). In resolution theorem proving, the classical source for the modelling of search is 3], which was not concerned with evaluating the complexity of the strategies. The primary objective of strategy analysis is to study the complexity of searching for a proof. An approach to this problem was proposed in 6]. It applies classical techniques from algorithm analysis to derive worst-case upper bounds ?