A TURING Test Scenario to Estimate an AI System's Validity

There is an urgent need to develop, to implement and to apply methodologies resp. tools for complex system validation. The inability to evaluate complex systems may become a crucial limiting factor to future technological progress. The authors are engaged in a quite comprehensive endeavour towards intelligent system validation. This paper deals with both the fundamentals of a so-called \TURING Test Methodology" for intelligent system validation and some ideas to make them practicable. First, we survey several concepts of veri cation and validation. Our favoured concepts are lucidly characterized by the words that veri cation guarantees to build the system right whereas validation deals with building the right system. Next, we critically inspect the thought-experiment called the TURING test. It turns out that, although this approach may not be su cient to reveal a system's intelligence, it provides a suitable methodological background to certify a system's validity. The principles of our validation approach are surveyed. The informal discussion leads to a collection of a few formalisms suitable for a systemtic approach to intelligent system validation. A discussion of essential problems beyond the basic TURING test methodology reveals some problems in making it practicable. Even in very simple scenarios the problem of vagueness and uncertainty in the experts' knowledge has to be managed in a reasonable way. This is due to the fact that the real target system's behaviour is not available as a yardstick. Instead, experts' knowledge has to be taken as a basis for system validation. The problem of measuring a system's behaviour against some knowledge source as insecure as human beings is analyzed. There is motivated, developed and illustrated a family of TURING test scenarios which provides the basis of further steps towards systematic intelligent system validation. The TURING test scenario presents some main ideas how to perform a (more or less) good approximation of the target knowledge by using some (more or less competent) experts' knowledge. Based on the precise formal concepts developed before, individual steps of performing the TURING test scenario can be described in some detail. The test results need to be evaluated to certify a given intelligent system's validity. Formal concepts can be utilized to estimate competence, local validity, and global validity.