A methodology for comparing grammar-based and robust approaches to speech understanding

We present a series of experiments designed to compare grammar-based and robust approaches to speech understanding, performed in the context of an Open Source medical speech translation system. We used two versions of the system, one grammar-based and one robust, trained off the same training data, and evaluated them on test data collected using both versions of the system. The experiments were constructed so as to avoid several methodological problems which occurred in earlier work reported in the literature. We found that the grammarbased version gave significantly better results than the robust version, with the difference increasing as subjects became more familiar with the system’s coverage. The rate of improvement in subject performance was positively affected by providing them with an intelligent online help system.