Captioning is the main method for accessing television and film content by people who are deaf or hard-of-hearing. One major difficulty consistently identified by the community is that of knowing who is speaking particularly for an off screen narrator. A captioning system was created using a participatory design method to improve speaker identification. The final prototype contained avatars and...

At the recent Alexander Graham Bell Association for the Deaf and Hard of Hearing (AG Bell) convention, I spoke up during a session supporting real-time captioning in classrooms. After all, I had recently finished my Ph.D. on this topic and knew of many studies validating their use. As soon as the session was over, I was asked by parents and educators for a reference to a specific study I had me...

A new real-time closed-captioning system for Japanese broadcast news programs is described. The system is based on a hybrid automatic speech recognition system that switches input speech between the original program sound and the rephrased speech by a ”re-speaker”. It minimises the number of correction operators, generally to one or two, depending on the difficulties of the speech recognition, ...

Video captions, also known as same-language subtitles, benefit everyone who watches videos (children, adolescents, college students, and adults). More than 100 empirical studies document that captioning a video improves comprehension of, attention to, and memory for the video. Captions are particularly beneficial for persons watching videos in their non-native language, for children and adults ...

Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images. In this paper we advocate for explicitly modeling the partial order structure of this hierarchy. Towards this goal, we introduce a general method for learning ordered representations, and show how it can be applied to a variety of tasks involv...

Automatic captioning of images is a task that combines the challenges image analysis and text generation. One important aspect notion attention: how to decide what describe in which order. Inspired by successes translation, previous works have proposed transformer architecture for captioning. However, structure between semantic units (usually detected regions from object detection model) senten...

Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representat...

The existing image captioning approaches typically train a one-stage sentence decoder, which is difficult to generate rich fine-grained descriptions. On the other hand, multi-stage image caption model is hard to train due to the vanishing gradient problem. In this paper, we propose a coarse-to-fine multistage prediction framework for image captioning, composed of multiple decoders each of which...

Abstract Diagnostic captioning (DC) concerns the automatic generation of a diagnostic text from set medical images patient collected during an examination. DC can assist inexperienced physicians, reducing clinical errors. It also help experienced physicians produce reports faster. Following advances deep learning, especially in generic image captioning, has recently attracted more attention, le...

It is well believed that the higher uncertainty in a word of caption, more inter-correlated context information required to determine it. However, current image captioning methods usually consider generation all words sentence sequentially and equally. In this paper, we propose an uncertainty-aware framework, which parallelly iteratively operates insertion discontinuous candidate between existi...