Decision Level Fusion of Visual and Acoustic Features of the Driver for Real-time Driver Monitoring System

Poor attention of drivers towards driving can cause accidents that can harm the driver or surrounding people. The poor attention is not only caused by the drowsiness of the driver but also due to the various emotions/moods (for example sad, angry, joy, pleasure, despair and irritation) of the driver. The emotions are generally measured by analyzing either head movement patterns or eyelid movements or face expressions or all the lasts together. Concerning emotion recognition visual sensing of face expressions is helpful but generally not always sufficient. Therefore, one needs additional information that can be collected in a non-intrusive manner in order to increase the robustness of the emotion measurement in the frame of a nonintrusive monitoring policy. We find acoustic information to be appropriate, provided the driver generates some vocal signals by speaking, shouting, crying, etc. In this paper, appropriate visual and acoustic features for emotion recognition applications are identified based on the experimental analysis. For visual and acoustic features, Linear Discriminant Analysis (LDA) technique is used for dimensionality reduction and Hausdorff distance is used for emotion classification. The performance is evaluated by using the different emotional recognition databases namely Indian face database for visual emotion recognition, Berlin database for acoustic emotion recognition and Vera am Mittag (VAM) database for both visual and acoustic emotion recognition. We propose a decision level fusion technique, to fuse the combination of visual sensing of face expressions and pattern recognition from driver’s voice. The result of the proposed approach is evaluated over the VAM database with various conditions.