This DISC Working Paper present a first outline of a systematic approach to the evaluation of spoken language dialogue systems (SLDSs) and their components. The approach is illustrated for the case of dialogue manager evaluation. The basic components of the approach are (a) a model of the core properties of the SLDS or component to be evaluated; (b) a set of evaluation criteria which have been ...

Regression testing of natural language systems is problematic for two main reasons: component input and output is complex, and system behaviour is context-dependent. We have developed a generic approach which solves both of these issues. We describe our regression tool, CONTEST, which supports context-dependent testing of dialogue system components, and discuss the regression test sets we devel...

We have designed and implemented a dialogue management design tool for use in dialogue design as a component of user-interface design in multi-modal applications. The tool provides: a formal language (typed feature structures) for describing states and events; a simple rule formalism for specifying dialogues; an automatic dialogue-property checking module; a dialogue-simulator for interactive t...

Natural Language Understanding Systems (NLU) will not be widely deployed unless they are technically mature and cost effective to develop. Cost effective development hinges on the availability of tools and techniques enabling the rapid production of NLU applications through minimal human resources. Further, these tools and techniques should allow quick development of applications in a user frie...

Spoken dialogue systems (SDS) allow users to interact with a wide variety of information systems using speech as the primary, and often the only, communication medium. The principal elements of an SDS are a speech understanding component which converts each spoken input into an abstract semantic representation called a user dialogue act, a dialogue manager which responds to the user’s input and...

In this paper we show how the notion of context has been reened in order to fullll the requirements posed by a natural language processing system. We describe the context model of the speech translation system verbmobil which has been derived from constraints given by various subcomponents of the system. In verbmo-bil context is stored in the dialogue memory which is incrementally constructed b...

This paper describes the integration of Amazon Alexa with the Talkamatic Dialogue Manager (TDM), and shows how flexible dialogue skills and rapid prototyping of dialogue apps can be brought to the Alexa platform. 1. Alexa Amazon’s Alexa 1 is a spoken dialogue interface open to third party developers who want to develop their own Alexa ”skills”. Alexa has received a lot of attention and has brou...

This paper describes an ISO project developing an international standard for annotating dialogue with semantic information, in particular concerning the communicative functions of the utterances, the kind of content they address, and the dependency relations to what was said and done earlier in the dialogue. The project, registered as ISO 24617-2 Semantic annotation framework, Part 2: Dialogue ...