A Multi-Modal Transformer network for action detection

This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our utilizes new attention mechanism that computes correlations between different spatial and motion modalities combinations. Exploring such has not been attempted previously. use modality more effectively, we suggest an algorithm corrects distortion caused by camera movement. Such distortion, common videos, severely reduces expressive power of features as optical flow fields. Our proposed outperforms state-of-the-art methods on two public benchmarks, THUMOS14 ActivityNet. We also conducted comparative experiments instructional activity dataset, including large set challenging classroom videos captured from elementary schools.