Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment

Image-text retrieval requires the system to bridge heterogenous gap between vision and language for accurate while keeping network lightweight-enough efficient retrieval. Existing trade-off solutions mainly study from view of incorporating cross-modal interactions with independent-embedding framework or leveraging stronger pretrained encoders, which still demand time-consuming similarity measurement heavyweight model structure in stage. In this work, we propose an image-text alignment module SelfAlign on top framework, improves accuracy maintains efficiency without extra supervision. contains two collaborative sub-modules that force at both concept level context by self-supervised contrastive learning. It does not require embedding during training maintaining independent image text encoders With comparable time cost, consistently boosts state-of-the-art non-pretraining models respectively 9.1%, 4.2% 6.6% terms R@sum score Flickr30K, MSCOCO 1K MS-COCO 5K datasets. The also outperforms most existing interactive-embedding orders magnitude decrease time. source code is available at: https://github.com/Zjamie813/SelfAlign.