In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal task...

In this paper, we describe our submissions to the WMT17 Multimodal Translation Task. For Task 1 (multimodal translation), our best scoring system is a purely textual neural translation of the source image caption to the target language. The main feature of the system is the use of additional data that was acquired by selecting similar sentences from parallel corpora and by data synthesis with b...

In this paper, we describe our submissions to the WMT17 Multimodal Translation Task. For Task 1 (multimodal translation), our best scoring system is a purely textual neural translation of the source image caption to the target language. The main feature of the system is the use of additional data that was acquired by selecting similar sentences from parallel corpora and by data synthesis with b...

Multimodal grammars provide an effective mechanism for quickly creating integration and understanding capabilities for interactive systems supporting simultaneous use of multiple input modalities. However, like other approaches based on hand-crafted grammars, multimodal grammars can be brittle with respect to unexpected, erroneous, or disfluent input. In this article, we show how the finite-sta...

Film and television works are multimodal discourse composed of a variety symbol systems such as text, sound image, so the audience's various senses can be mobilized at same time when watching movies. Starting from perspective analysis, this paper applies Delu Zhang’s theoretical framework analysis to analyze subtitle translation Harry Potter Philosopher's Stone four aspects: culture, context, c...

Recently, there has been a surge in research multimodal machine translation (MMT), where additional modalities such as images are used to improve quality of textual systems. A particular use for systems is the task simultaneous translation, visual context shown complement partial information provided by source sentence, especially early phases translation. In this paper, we propose first Transf...

We propose word-region alignment-guided multimodal neural machine translation (MNMT), a novel model for MNMT that links the semantic correlation between textual and visual modalities using alignment (WRA). Existing studies on have mainly focused effect of integrating modalities. However, they do not leverage relevance two advance in by incorporating WRA as bridge. This proposal has been impleme...

This work discusses the results of two user studies aiming to evaluate the NESPOLE! speech-to-speech translation system, which provides for multilingual and multimodal communication in the tourism and in the medical domain, allowing users to interact through the Internet by sharing maps, web-pages and pen-based gestures. The purpose is to investigate the overall effectiveness of the combination...