Learning Intuitive Physics with Multimodal Generative Models

Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual tactile feedback make predictions about expected motion in dynamic scenes. Visual information captures object properties such as 3D shape location, while provides critical cues forces resulting it makes environment. Utilizing novel See-Through-your-Skin (STS) sensor high resolution multimodal sensing surfaces, our system both appearance objects. We interpret dual stream signals from using Multimodal Variational Autoencoder (MVAE), allowing us capture modalities contacting develop mapping vice-versa. Additionally, perceptual can be used infer outcome physical interactions, which we validate through simulated real-world experiments resting state an predicted given initial conditions.