LGANet: Local and global attention are both you need for action recognition

Due to redundancy in the spatiotemporal neighborhood and global dependency between video frames, recognition remains a challenge. Some prior works have been mainly driven by 3D convolutional neural networks (CNNs) or 2D CNNs with well-designed module for temporal information. However, convolution-based lack capability capture due limited receptive field. Alternatively, transformer is proposed build long-range frame patches. Nevertheless, most transformer-based significant computational costs because attention calculated among all tokens. Based on these observations, we propose an efficient network which dub LGANet. Unlike conventional transformers recognition, LGANet can tackle both learning local token affinity shallow deep layers, respectively . Specifically, implemented layers reduce parameters eliminate redundancy. In spatial-wise channel-wise self-attention are embedded realize of high-level features. Moreover, several key designs made multi-head (MSA) feed-forward (FFN). Extensive experiments conducted popular benchmarks, such as Kinetics-400, Something-Something V1&V2. Without any bells whistles, achieves state-of-the-art performance. The code will be released soon.