Multimodal User Interface Design Increasing Accessibility to Mathematics For Blind students

The project investigates the potential benefits of multimodal user interfaces for blind students. A multimodal user interface uses the combination of two or more different human senses to interact with computers. The work focused on finding the needs of the interaction model for blind students by using different human modalities, so that they perceive and control information in efficient and effective ways. Sweller’s cognitive load theory (Sweller 1988) is applied in this study to examine efficient and effective learning through computer interaction . In the first part of this work, we studied how visually impaired computer users interact with computers with currently used solutions, screen reader and Braille display. In the second part, the work explored the potential use of haptic technology to visualize the computer information for mathematical learning as an experimental data type which visually impaired people have difficulty to access. The work was carried out in a real context with users who are blind. The result indicated that the audio user interface (screen-reader) was not always the best interaction solution though this is the most commonly used among the blind computer users. The effectiveness of tactile user interface (Braille-display) to accomplish certain tasks was similar to the audio user interface, while the users' satisfaction rate was higher. Haptic user interface was tested with a prototype application developed for this study on a haptic machine (PHANToM). In the haptic experiment, we have focused on the accuracy of information perceived by users through haptics. Users were able to guess the shape of virtual object but identifying the details of the object with the prototype is not accurate for the purpose of mathematical learning use. At this point, we concluded that haptic-only perception has a limitation to interact with computer. We presented the effects of the single mode as a result of this work. It is important to understand how blind students interact with computer through the single mode in order to develop the effective and efficient multimodal user interface. By understanding the effect of each mode, we are able to optimize the use of each mode when combining them together. The study analyzed the recommendations and open questions directed to audio (screen reader) and tactile (Braille display) user interfaces. The further research is required to investigate the strength of haptic interaction. We have presented the suggestions of further research and hypothesis that needs to be evaluated in the future work.