Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs

Figure skating scoring is challenging because it requires judging players’ technical moves as well coordination with the background music. Most learning-based methods struggle for two reasons: 1) each move in figure changes quickly, hence simply applying traditional frame sampling will lose a lot of valuable information, especially 3 to 5 minutes lasting videos; 2) prior rarely considered critical audio-visual relationship their models. Due these reasons, we introduce novel architecture, named Skating-Mixer. It extends MLP framework into multimodal fashion and effectively learns long-term representations through our designed memory recurrent unit (MRU). Aside from model, collected high-quality FS1000 dataset, which contains over 1000 videos on 8 types programs 7 different rating metrics, overtaking other datasets both quantity diversity. Experiments show proposed method achieves SOTAs all major metrics public Fis-V dataset. In addition, include an analysis recent competitions Beijing 2022 Winter Olympic Games, proving has strong applicability.