Moving towards the Assessment of Collaborative Problem Solving Skills with a Tangible User Interface

The research on the assessment of collaborative problem solving (ColPS), as one crucial 21 Century Skill, is still in its beginnings. Using Tangible User Interfaces (TUI) for this purpose has only been marginally investigated in technology-based assessment. Our first empirical studies focused on light-weight performance measurements, usability, user experience, and gesture analysis to increase our understanding of how people interact with TUI in an assessment context. In this paper we propose a research agenda for assessing ColPS of individuals using the Microworlds methodology implemented on TUIs. In a first example item, we use so-called MicroDYN items, which are independent microworld scenarios that rely on structural linear equations as underlying model. As the MicroDYN approach has been thoroughly empirically investigated for the assessment of complex problem solving of individuals, it offers a good basis for a reliable and valid assessment. We describe how this approach was applied to create an assessment item for a collaborative setting. This item described in this paper implements a simplified model of a MicroDYN item related to climate change using knowledge of previous studies. Therefore, the focus of the item’s construction lies on meeting the requirements for a standardised high quality assessment. Finally, a research agenda is proposed to sketch the main research issues. INTRODUCTION In the last few years the term 21 Century Skill gained substantial visibility in scientific literature, for instance in the latest reports on technology-based assessment (OECD, 2012) as well as in the Digital Agenda published by the European Commission (Ferrari, 2012). These so-called 21st Century Skills refer to skills such as complex problem solving, collaborative problem solving, creativity, critical thinking, learning to learn, decision making, etc. (Binkley et al., 2012). Several researchers have stated that the acquisition of these 21 Century Skills and their development are only marginally investigated (Bennett & Gitomer, 2009) despite their unquestioned importance and relevance. This paper focuses on a particular 21st Century Skill, collaborative problem solving (ColPS), which encompasses the ability to successfully deal with untransparent and dynamically changing problems in a collaborative setting. ColPS is considered an important component for the success in life and for this reason will be included as transversal domain in one of the most prestigious large-scale assessments worldwide, the Programme for International Student Assessment (PISA) in its 2015 cycle (OECD, 2014). As emphasised by educators and policy makers, ColPS plays an important role in different life contexts at schools, at home, or at work (Rummel & Spada, 2005). Especially on the labour market, the organisational researchers (e.g. (CannonBowers & Salas, 1997; Hellinghausen & Myers, 1998)) emphasise more and more the importance of teamwork for organisational success and continuously look for new ways to assess collaborative skills in a recruitment process and in the course of personnel development. Despite the apparent importance of ColPS for success in different areas of life, up until now there has not been much progress in its assessment. However, recently, fast development of new technologies for gathering data in real settings allows us to assess specific 21 Century Skills such as ColPS. Due to practical limitations, conventional paper-pencil tests or even the use of desktop applications known from computer-based assessment do not entirely allow for highly dynamic and interactive assessment of 21 Century Skills. More natural forms of interTOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 96 action such as touch, speech, gestures, and handwriting support assessment researchers to explore new approaches to skill assessment. Therefore, the availability of affordable devices that can collect such natural forms of interaction motivated us to develop several so-called tangible user interfaces (TUI) for assessment. ColPS assessment. The use of TUI in assessment is new and, therefore, knowledge about how to design those systems and how to develop test items for TUI is limited. For this reason, we present the state of the art in related research topics and propose a study that will help us shed light on the use of TUIs for ColPS assessment purposes. The aim of this article is to summarise open research issues in the domain of complex collaborative problem solving and to motivate the use of tangible user interfaces to assess this 21 Century Skill. The upcoming section summarises research on relevant topics followed by the description of the overall research methodology. Further, we elaborate how a test item related to climate change has been developed for the TUI to assess collaborative problem solving of individuals. Finally we derive a first research agenda to assess complex collaborative problem solving with TUIs. BACKGROUND Collaborative Problem Solving ColPS is one of the 21 Century Skills that has attracted the most attention from researches in recent years. This comes from the fact that collaborative work is becoming ever more significant in education and in everyday life. For instance, in schools students are often required to work in teams on science tasks; at work people collaborate on common projects; and at home families take household decisions collaboratively. An everyday example of ColPS is students organising a school trip together. They need to consider different factors such as distance, transport, or price, whereby some of factors can change in the course of planning. Students take over specific roles in the process (e.g. a person who finds bus schedules, the one taking care of hotel prices, the one who makes an overview of everything), exchange their knowledge and apply different strategies to find the nearest destination, with the optimal transport and the cheapest price. In theory, ColPS incorporates two dimensions – complex problem solving as the cognitive dimension and collaboration as the interpersonal dimension. However, ColPS is not just the sum of those two dimensions, but represents the interaction of complex problem solving skills and collaboration skills (OECD, 2014). The first dimension, complex problem solving, is one of the 21 Century Skills that has been extensively investigated. It may be defined as searching for the path from the problem state to the goal state. Its main characteristics are that the problem the person is confronted with is untransparent and the problem situation is dynamic, meaning that it changes, for instance, in the course of time or under the influence of the problem solver (Fischer, Greiff, & Funke, 2011; Lai, 2011). By adding the collaborative aspects to the complex problem solving construct leads us to the definition by O`Neil et al. (2003), who define ColPS as searching for the path from the initial state to the goal state while interacting with others working on a shared goal. Further, one of the most recognised large-scale assessments worldwide, the Programme for International Student Assessment (PISA) offers a more detailed definition of ColPS describing it as: "...the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution" (OECD, 2013). Whereas the dimensions of ColPS, problem solving and collaboration, are well examined as single constructs, the ColPS as a construct that incorporates both of those skills is still insufficiently investigated. Although recently there are initiatives researching the assessment of ColPS, especially in large scale testing context, the progress is still scarce at best. As outlined by Krkovic et al. (Krkovic, Pásztor-Kovács, Molnár, & Greiff, 2014) following aspects need to be considered when constructing the assessment for ColPS how to create a standardised testing environment, to use computer agents or humans as collaborators, how to assess communication, and how to analyse the collected data. Up until now, there have been two approaches on how to assess ColPS – human-to-agent and human-to-human. The first approach incorporates human-to-agent collaboration, the participant collaborates with a computer-simulated agent to solve a problem collaboratively. The OECD will include the assessment of ColPS in its PISA 2015 cycle uses this approach. The application of human-to-agent approach can be justified by the fact that it offers standardised stimuli to each participant ensuring a standardized testing environment and reasonable scoring procedures (Graesser, Jeon, & Dufty, 2008). Another approach to the assessment of ColPS is to use human-to-human interaction as investigated in the ATC21s project (Griffin, Care, & TOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 97 McGaw, 2011) which in comparison to the human-to-agent approach offers more face valid situation of two or more individuals working together on a problem, providing more in-depth information about the collaboration process. However, the human-to-human approach has its limitations too, such as unstandardized assessment settings, or large log files coming from open conversations that can hardly be transferred into scoring (O’Neil, et al., 2003). Considering benefits and limitations of both approaches, researchers need to search for an optimum solution for ColPS assessment that will on one hand offer scenarios that are scoreable and on the other hand be realistic and close to real life situations. Programs such as PISA aim at large scale assessment, hence, the technology they use must be available in schools and can be implemented in classes. For this reason PISA uses computer-based assessment which allows for the administration in school to use their existing IT infrastructure. Due to the novelty and low deployment rate of TUI, their use for large scale assessment seems unpractical in the near future. As of now, assessment on TUI could be used in scenarios such as personnel selection in human resources management. This scenario is quite probable considering that, at the moment, there is no reliable methodology to assess ColPS skills in the course of the selection process, where skills such as team work play an important role. Moreover , by using innovative user interfaces instead of computer-based testing richer data sets can be collected. TUI-based assessment The aim of tangible user interfaces (TUI) is to make computer bits tangible and to allow users to grasp and manipulate them with their hands (Ishii, 2008). This concept has a number of advantages for social and contextual interactions, such as collaboration (Hornecker & Buur, 2006). To date, literature provides first exploratory results on the learning benefits of TUIs, however, in the field of assessment research is still in its infancy. In particular, the additional haptic dimension, the better accessibility (for instance for children), and the shared space that can be used in collaborative situations (Marshall, 2007) are claimed to be beneficial in learning situations. According to Klemmer et al. (2006), the physical objects and actions of TUIs allow using multiple senses of our human bodies and have an essential impact on our understanding of the world. They encourage rapid epistemic actions (i.e. try and error actions)and thus lower cognitive load by simplifying thinking processes (Esteves, Van den Hoven, & Oakley, 2013). Learning and assessment can be supported by representing problems in a new way, using physical and digital elements. Further, the way the users are solving tasks can be detected and feedback can be directly given. Nevertheless, no TUI has been systematically used and evaluated in the context of technology-based assessment (TBA). Since research on using TUIs for assessment purposes is still at the very beginning, Ras et al. (2013) have conducted a series of empirical studies in order to identify harmonies and tensions concerning the use of TUIs for assessing skills of complex and collaborative problem solving in a low-stake assessment context. The aims were manifold: at a first stage it was interesting to observe how users interact with the table and the tangibles in general and which spaces are used for interaction, communication, and collaboration. Furthermore, each study also assessed the user experience and the usability of the device to ensure a continuous improvement cycle. The figures below show already two different devices. Several items have been developed, two of which are depicted in Fig. 1 respectively Fig 2. The aim of the matching item was to assign the labels of the planets (put on small tangibles) to the images. The goal of the windmill item was to explore the impact of different variables such as wind speed, number of blades, and height of the windmill on the energy produced by the windmill. The output changed in real-time in response to the manipulations of the input. TOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 98 Fig. 1. Matching item Fig. 2. Simulation item A description of the outcomes of these studies can be found in Ras et al. (2013). In summary, it was interesting to observe that different spaces where used to intensively communicate using gestures, talk, and physically manipulating objects. Besides the interactive surface also the space between participants or even the non-reactive border of the table was used to solve the item (e.g., to sort, group, or even exclude tangibles). The large space of the TUI allows for more than one person to get actively involved. But the position at the table also reveals that a person might feel responsible for a specific space (or even tangibles) or cannot access other spaces because of the distance or the closeness to another participant. We observed that all activities across space support the participants to acquire knowledge about the problem situation and apply the acquired knowledge, which are two important dimensions of ColPS (Funke, 2001; Greiff et al., 2013). Another requirement for an assessment context is to provide an authentic and intuitive environment. Both the windmill item as well as the climate change item described in the next section represent real-world scenarios. Further, the physical tangibles allow the participants to manipulate the parameters in a natural and simple way. They enable more efficient interactions, such as turning a parameter without looking at it, while another parameter can be observed on the TUI surface. These activities and many others we observed help the participants to acquire knowledge about the problem situation. In addition, during the next step where the participants apply knowledge to solve the problem other patterns have been detected such as: users often hold them in their hands, use them to make gestures and suggestions, or they produced noise with the object to increase awareness in the group. Well-designed tangibles may implement a specific metaphor to increase its understanding but we cannot ensure that users immediately understand how to us or manipulate it. Holding objects in their hand can harm other users to contribute or even impact the complete solving strategy. TOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 99 During the first step of the problem solving process, knowledge on the problem situation needs to be acquired by all participants. We have observed that, in particular, during a first phase several uncoordinated trial and error manipulations were done, even in parallel by several participants at once, followed by participants suggesting hypotheses of how parameters impact others without any deeper reflection. Hence, the challenge is certainly to distinguish these chaotic, unplanned activities from those that are more systematic and serve to understand the problem or even to solve the problem. For each study we interviewed assessment experts with respect to the item and the usefulness of the table for assessment. One interesting outcome was that they claim that a TUI supports the users in recognizing and understanding the perspectives of others – an important skill of collaborative problem solving. Gestures and spoken communication support them to easily explain hypotheses. Different perspectives of individuals (e.g., a misunderstanding) can be recognized through realising them on the table. In assessment is it clear that the technology used for assessment can impact test performance. Aspects such as motivation, familiarity with a technology, even anxiety to use a specific technology need to be measured as disturbing variables. User experience is a typical measure to evaluate software and its measurement serves to improve both pragmatic and hedonic goals of the user. Unfortunately, all user experience and usability scales have been developed for classical software but not for applications running on tangible devices. They do not explicitly distinguish between physical and digital characteristics of TUI-based applications. Therefore, a research objective of these studies was also to investigate weaknesses of standardised measurement instruments tailored to evaluate usability and user experience. A first study revealed weaknesses of the Simple Usability scale (Ras & Maquil, 2011) and a new measurement instrument, which combines measures of several existing models, was developed and evaluated with the Windmill example (Ras, et al., 2013). The windmill item is similar to a simulator: parameters can be manipulated and impacts on other output parameter can be observed. In the following section so-called MicroDYN items have been chosen as a task type for the collaborative problem solving scenario to be implemented on a TUI. MICRODYN AS TASK FOR COLPS – A CONCRETE EXAMPLE MicroDYN This study is the first to suggest implementing so-called MicroDYN items on a TUI and to employ them in a group (i.e., a collaborative) setting in order to assess ColPS. Specifically, MicroDYN tasks are independent microworld items, which are based on structural equations. Until recently, MicroDYN items have only been used as a computer-based assessment approach towards assessing complex problem solving skills in individuals. Such an approach enables very well to control the difficulty and complexity of items, as well as the scoring procedures. In MicroDYN, the problem solver can manipulate a set of input variables, which reflects on specific output variables. Thereby, the problem solver needs to acquire knowledge about input and output variables and detect causal relations between them by representing it in a graphic model (dimension of knowledge acquisition, (Mayer & Wittrock, 2006)). In a subsequent step, the problem solver needs to apply the acquired knowledge in order to reach predefined goals (dimension of knowledge application, (Novick & Bassok, 2005)). Several studies have shown the validity of MicroDYN with regard to the assessment of complex problem solving (e.g. (Greiff, et al., 2013)). Recently, MicroDYN items have been implemented into the human-to-agent computer-based ColPS assessment. In such tasks, the participant has limited controls over the input variables and must collaborate with a computersimulated agent in order to acquire knowledge and apply it in the further step. Thereby, the participant can communicate with the agent by using a chat with predefined messages. In this study we expand this approach by implementing MicroDYN in the collaborative setting by using human-to-human approach, whereby the participants work together on tangible table interface. Moreover, the use of tangible table interface for the first time for ColPS assessment will incorporate triads instead of dyads that are used in computer-based version of collaborative MicroDYN tasks, which will enrich the collaboration aspect. A tangible climate change item Climate change as a complex problem Today, scientists agree that the Earth’s climate system is warming up and that a significant cause of the change is due to human activities (IPCC Working Group I, 2013). This is often referred to as Global Warming. Global Warming is mainly evidenced by increases in global average air and ocean temperatures resulting in the widespread melting of snow and ice leading to rising global average sea level. The main reason of this warming is caused by increases in concentrations of greenhouse gases, to which the largest contributor is carbon dioxide (CO2). Certain waste management and agricultural practices aggravate the problem by releasing chemical compounds such as methane and nitrous oxide, which further fuel the warming, trapping ever more heat in the atmosphere. TOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 100 To allow children to understand the relations between the different variables, as well as to follow the MicroDYN methodology of Greiff et al. (2013), a simplified model of climate change was defined. The model’s principle is as follows. The system is configured with eight variables, each with three sub-variables (see Fig. 3) and their respective simplified impact on the total amount of CO2 (i.e., approximated on the basis of (klimAktiv gemeinnützige Gesellschaft zur Förderung des Klimaschutzes mbH, 2014)). The variables allow children to explore the impact on global CO2 emissions using examples they are familiar with such as meat consumption, transportation means to school respectively to their vacation destination, or simply the amount of clothing they own. By selecting one of the sub-variables which implement, in various forms, the high, medium, and low choices for the given variable, they express their choice. The system shows the effect on the different outputs as given by the MicroDYN methodology. The feedback is visual and based on a known scenario the children were confronted with prior to the experiment as part of their curriculum on climate change (such as the dire situation of polar bears). Children are then able to freely change their assignment of variables to explore the magnitude of variables on CO2 emissions. Collaborative widgets for manipulating variables To allow for a maximum of simultaneous accessibility and shareability of controls we designed a novel type of widgets: the Tangible Pie Widget. Questions are visualized as immutable pie charts (Ø 15cm) with physical borders where each of the sections corresponds to one potential answer. Prior to the solving activity, each of the children gets one token per question. The children can place their token into one of the sections in order to give their personal answer to the provided question. This allows defining the weights of the different sections of the pie widget that can then be translated to the world population (see Fig. 3). Fig 3. The tangible pie widget allows multiple users to change the weights of the sections by moving small tokens For each parameter, the system provides feedback by depicting the value (see Figure XY) . Further it calculates and visualises the output of the total amount of CO2 emissions and global temperature as defined by all pie widgets (see Figure 4). The pie widgets can be used in two different ways. At the beginning, they are empty. Children place their token into the slices of the pie widget to cast their vote for each question. The system calculates the CO2 output and the impact on global temperature in real time as votes are cast. This allows them to become familiar with the tool and to define the initial state. Furthermore, children can explore WHAT-IF scenarios, that is, they can change the value of variables by placing the tokens in a different section. This approach incites all children to actively participate in setting the values and, in the second phase, allows them to collaborate in small groups to simultaneously manipulate the widgets. Exploring different scenarios will allow children to identify what behaviour has the most impact on CO2 output and hence acquire knowledge about the underlying model. TOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 101 Fig 4. Feedback provided by the system Fig 5. The system in use by a group of children System architecture and design The system features two main components: the TUI library (i.e., TULIP) that allows defining and instantiate all kind of tangibles; and a library that allows for defining complex problems. The TUI library is used to define all feedback and interactive tangibles as shown in Fig. 3 provided to the children. The complex problem solving library provides mathematical models such as linear equations which can be instantiated, for example, as seen in Equation 1 and 2. Each equation of the system is fed by the variables of one question while an instance of Equation 2 is used to provide the global CO2 output.