Minus-Two: Multimedia, Sound Spatialization and 3D Representation for Cognitively Impaired Children

Multimedia and Hypermedia technologies can be successfully used in education and rehabilitation for cognitively disabled persons. In particular, storytelling has always proved successfully in creating imaginary worlds that could both protect the child from the difficulties of the outer world and create the conditions to feed new stimuli without causing discomfort. However, traditional storytelling techniques (paper, pencil, the voice of the teacher) sometimes might be vastly improved by using multimedia and hypermedia technologies. The possibility to play different music, to explore spatialized sounds, to record sounds and voices, and the opportunity to play with 3D images within a 3D space using devices as simple as touch screens, may ensure an enhanced level of attention from the children and may give to the teacher new and powerful tools to convey his/her message through the common activity. In this paper we motivate the use of hypermedia educational techniques with cognitively impaired children, outlining the general requirements of an educational framework that might be provided to teachers. We then describe the functioning of a prototype we developed following these requirements, MINUS-TWO: a storytelling environment mainly characterized by a strict cooperation between the child and the teacher or the therapist, by multimodal interaction, sound spatialization and 3D representation. The paper also presents the results we obtained by using these techniques with more than twenty cognitively impaired children. 1 Foreword and Motivations: Storytelling as an Activity Towards Autonomy Marc Augè [1], in his book published in 1993, has introduced the concept of nonlieu (non-place), defining it as “a space in which who passes through cannot read anything about his identity, nor about the relation with himself, the relations with others, or more generally, about relations among ones and others, or a fortiori about their common background”. This article, in agreement with a rich literature, proposes narration as affirmative answer to non-lieu: the place of the meeting, the communication and the refinement of different and growing identities. Narration can be the place where identity connects to other paths and a person can perceive his 2 Thimoty Barbieri, Antonio Bianchi, and Licia Sbattella origins and his possible developments. The expression of emotive and affective aspects of situated knowledge founds in narration its natural vehicle. The main motivation for the storyteller framework described in this paper is the following: narration is a fundamental opportunity for a human being to develop and maintain his personal identity. An opportunity which cannot be overlooked especially in cases of cognitive disability, even when language or relational competencies are strongly compromised (think for example to autistic syndromes). A person in situation of disability risks to be condemned to live solely in physical environments and to be taken apart from a representation based environment; he risks namely to be overwhelmed by his immediate needs, a condition which compromises his harmonious development. A story organizes the human space in different ways; creating complex time sequences it allows to explore different situations, and to face them with more flexibility. A dramatization may be useful to let the students understand a situation by playing it, moving it to a different plane, retrieving only in a second moment its details connected to reality. Building a tale as a background creates a sort of perceptive structure. It is at the same time a recognition of a given reality, it is a given "data" from which the single actor may receive support and stimuli. We can think of a storyteller framework as a rehabilitation tool too, in the sense it offers stimuli directed to the sphere of attention, memory, problem solving, both from a logical-deductive and relational point of view. Situations characterized by joy, sadness, anxiety, may be faced in an interactive way cooperating with the tutor in known contexts, under his supervision. The literature describes different storytelling activities involving persons with different kind of problems (e.g. Alzheimer...). Pedagogy and traditional rehabilitation often use objects such as dolls, toys and books to support narration or encourage children to draw using paper and colored pencils. Why then using an information technology framework for storytelling? It is very difficult to involve in a storytelling activity children who don’t speak and do not draw, who feel this kind of relationship really uncomfortable. We think that 3D spaces, sounds and music may improve opportunities contributing to stimulate attention and to make the experience more vivid. Beside this, it gives the important possibility to introduce variations, to create new scenarios, to support different views on narration, to allow the listener to interact with the story as it gets unfold, and with the description of characters as they present themselves, as a further step in stimulating the child to an autonomous elaboration of information and messages proposed by the teacher. Within a 3D scene, the sound acquires a spatial dimension, and is perceived as directly linked to particular objects. Navigating in the space, it is possible to listen to how sounds change in direction and intensity, adding the possibility to explore the objects even from this point of view. The possibility to interact with the environment in different ways using traditional interface devices, touch screens, voice or real musical instruments – gives more opportunities to involve children with cognitive, expressive or relational disabilities, as shown by the use of our prototype with more then twenty cognitively impaired and autistic children. Minus-Two: Multimedia, Sound Spatialization and 3D Representation for Cognitively