Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning

One of the key factors enabling machine learning models to comprehend and solve real-world tasks is leverage multimodal data. Unfortunately, annotation data challenging expensive. Recently, self-supervised methods that combine vision language were proposed learn representations without annotation. However, these often choose ignore presence high levels noise thus yield sub-optimal results. In this work, we show problem estimation for can be reduced a density task. Using estimation, propose building block representation based strictly on inherent correlation between different modalities. We demonstrate how our broadly integrated achieves comparable results state-of-the-art performance five benchmark datasets two tasks: Video Question Answering Text-To-Video Retrieval. Furthermore, provide theoretical probabilistic error bound substantiating empirical analyze failure cases. Code: https://github.com/elad-amrani/ssml.