An Emotion Recognition Method Based on Eye Movement and Audiovisual Features in MOOC Learning Environment

In recent years, more and people have begun to use massive online open course (MOOC) platforms for distance learning. However, due the space–time isolation between teachers students, negative emotional state of students in MOOC learning cannot be identified timely. Therefore, receive immediate feedback about their states. order identify classify learners’ emotions video scenarios, we propose a multimodal emotion recognition method based on eye movement signals, audio images. this method, two novel features are proposed: feature coordinate difference eyemovement (FCDE) pixel change rate sequence (PCRS). FCDE is extracted by combining trajectory optical flow trajectory, which can represent learner’s attention degree. PCRS from image, speed image switching. A extraction network convolutional neural (CNN) (FE-CNN) designed extract deep three modals. The inputted into classification CNN (EC-CNN) emotions, including interest, happiness, confusion, boredom. single modal identification, accuracies corresponding modals 64.32%, 74.67%, 71.88%. fused feature-level fusion, decision-level model-level fusion methods, evaluation experiment results show that achieved highest score 81.90% recognition. Finally, effectiveness FCDE, FE-CNN, EC-CNN modules verified ablation experiments.