Partial Information in Multimodal Dialogue

Much research has been directed towards developing mul-timodal interfaces in the past twenty years. Many current multimodal systems, however, can only handle multimodal inputs at the sentence level. The move towards multimodal dialogue signiicantly increases the complexity of the system as the representations of the input now range over time and input modality. We developed a framework to address this problem consisting of three parts. First, we propose to use multidimen-sional feature structures, a straightforward extension of typed feature structures, as a uniform representational formalism in which the semantic content stemming from all input modalities can be expressed. Second, we extend the feature structure formalism by an object-oriented framework that allows the back-end application to keep track of the state of the representation under discussion. And third, we propose an informa-tional characterization of dialogue states through a constraint logic program whose constraint system consists of the multidimensional feature structures. The multimodal dialogue manager uses the characterization of dialogue states to decide on an appropriate strategy.