Semantics-Aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition

Existing vision-based action recognition is susceptible to occlusion and appearance variations, while wearable sensors can alleviate these challenges by capturing human motion with one-dimensional time-series signal. For the same action, knowledge learned from vision sensors, may be related complementary. However, there exists significantly large modality difference between data captured wearable-sensor vision-sensor in dimension, distribution inherent information content. In this paper, we propose a novel framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), enhance (videos) adaptively transferring distilling multiple sensors. The SAKDN uses wearable-sensors as teacher modalities RGB videos student modality. To preserve local temporal relationship facilitate employing visual deep learning model, transform signals of two-dimensional images designing gramian angular field based virtual image generation model. Then, build Similarity-Preserving Multi-modal Fusion Module fuse intermediate representation different networks. Finally, fully exploit transfer well-trained networks network, Graph-guided Semantically Discriminative Mapping loss, which utilizes graph-guided ablation analysis produce good explanation highlighting important regions across concurrently preserving interrelations original data. Experimental results on Berkeley-MHAD, UTD-MHAD MMAct datasets well demonstrate effectiveness our proposed SAKDN.