We address the problem of jointly learning vision and language to understand the object in a fine-grained manner. The key idea of our approach is the use of object descriptions to provide the detailed understanding of an object. Based on this idea, we propose two new architectures to solve two related problems: object captioning and natural language-based object retrieval. The goal of the objec...

Closed captioning has been enabling access to television for people who are deaf and hard of hearing since the early 1970s. Since that time, technology and people’s demands have been steadily improving and increasing. Closed captioning has not kept up with these changes. We present the results of a study that used graphics, colour, icons and animation as well as text, emotive captions, to captu...

Previous models for video captioning often use the output from a specific layer of a Convolutional Neural Network (CNN) as video features. However, the variable contextdependent semantics in the video may make it more appropriate to adaptively select features from the multiple CNN layers. We propose a new approach for generating adaptive spatiotemporal representations of videos for the captioni...

Working memory (WM) may be an essential component of incidental vocabulary learning and retention from captioned videos. However, how WM affects young learners’ under different types captions remains unclear. The present study employs a between-subject research design. main purpose is to examine two WM— phonological short-term complex WM—impact outcomes incidentally learned retained three capti...

Image captioning using deep neural networks has recently gained increasing attention, mostly for English langue, with only few studies in other languages. Good image model is required to automatically generate sensible, syntactically and semantically correct captions, which turn requires good models both computer vision natural language processing. The process more challenging case of data scar...

Image Captioning has gained tremendous spotlight in recent years. However, the captioning models generate captions English language. In this paper, we present an image caption generator for our regional language that is Hindi using Resnet50 and LSTM with attention module. An experimental study shown highlighting effect of attention-based learning on generated captions. Flickr8k dataset used to ...

Understanding concepts in the world remains one of the well-sought endeavours of ML. Whereas ImageNet enabled success in object recognition and various related tasks via transfer learning, the ability to understand physical concepts prevalent in the world still remains an unattained, yet desirable, goal. Video as a vision modality encodes how objects change across time with respect to pose, pos...

Evaluation metrics for image captioning face two challenges. Firstly, commonly used metrics such as CIDEr, METEOR, ROUGE and BLEU often do not correlate well with human judgments. Secondly, each metric has well known blind spots to pathological caption constructions, and rulebased metrics lack provisions to repair such blind spots once identified. For example, the newly proposed SPICE correlate...