Measuring Team Knowledge During Skill Acquisition of a Complex Task

The design of team training programs and other team interventions could benefit from an understanding of team cognition. The research presented in this article evaluates methods for eliciting and assessing team knowledge during acquisition of a complex task. Knowledge measures are evaluated in terms of their ability to predict team performance and also in terms of how they reflect skill acquisition. The study was conducted in the context of a synthetic three-person team task that involved operating an uninhabited air vehicle. Eleven teams of three Air Force ROTC cadets participated in three experimental sessions lasting from three to six hours. During these sessions teams were trained on the task and were observed as they performed ten 40-minute missions. During the missions, team performance and team process behaviors were measured, as well as the fleeting team knowledge associated with situation awareness. In addition, long-term team knowledge regarding both taskwork and teamwork were measured off-line in four sessions. Results indicated that teams reached asymptotic performance on this task after 1.5 hours of individual training and four 40-minute team missions. This skill acquisition was paralleled by improvements in team situation models, teamwork knowledge and to a lesser extent, team process behaviors. Taskwork relatedness ratings measured at both the individual and team level were good predictors of team performance and indicated that high performing teams had more knowledge of the task from the perspective of other team members, as opposed to lower performing teams. These measures help reveal the knowledge underlying team behavior, and thus have implications for team training and other interventions. Measuring Team Knowledge 3 MEASURING TEAM KNOWLEDGE DURING SKILL ACQUISITION OF A COMPLEX TASK Technological developments in the military and elsewhere have transformed highly repetitive manual tasks, requiring practiced motor skills into tasks that require cognitive skills often related to overseeing new technology such as monitoring, planning, decision making, and design (Howell & Cooke, 1989). As a result, a full understanding of many tasks, at a level required to intervene via training or system design, requires an examination of their cognitive underpinnings. Additionally, the growing complexity of tasks frequently surpasses the cognitive capabilities of individuals and thus, necessitates a team approach. For instance, teams play an increasingly critical role in complex military operations in which technological and informational demands necessitate a multioperator environment (Salas, Cannon-Bowers, Church-Payne, & Smith-Jentsch, 1998). Whereas the team approach is often seen as a solution to cognitively complex tasks, it also introduces an additional layer of cognitive requirements that are associated with the demands of working together effectively with others. Team members need to coordinate their activities with others who are working toward the same goal. Team tasks often call for the team to detect and recognize pertinent cues, make decisions, solve problems, remember relevant information, plan, acquire knowledge, and design solutions or products as an integrated unit. Therefore, an understanding of team cognition, or what some have called the new "social cognition" (Klimoski & Mohammed, 1994), is critical to understanding much team performance and intervening to prevent errors or improve productivity and effectiveness. Measuring Team Knowledge 4 The assessment and understanding of team cognition (i.e., team mental models, team situation awareness, team decision making) requires psychometrically sound measures of the constructs that comprise team cognition. However, measures and methods targeting team cognition are sparse and fail to address some of the more interesting aspects of team cognition (Cooke, Salas, Cannon-Bowers, & Stout, 2000). As measures of team cognition are developed, they can be used to better understand team cognition. The research presented in this article evaluates methods for eliciting and assessing team knowledge during acquisition of a complex task. Knowledge measures are evaluated in terms of their ability to predict team performance and also in terms of how they reflect skill acquisition. The knowledge measures that are evaluated here attempt to address some of the shortcomings of the current methodologies. In the following section team knowledge is defined in terms of a framework that also identifies some of these shortcomings. Team Cognition and Team Knowledge Salas, Dickinson, Converse, and Tannenbaum (1992) define team as "a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership" (p. 4). Thus, teams, unlike some groups, have differentiated responsibilities and roles (Cannon-Bowers, Salas, & Converse, 1993). This division of labor is quite common and enables teams to tackle tasks too complex for any individual. Measuring Team Knowledge 5 Interestingly, this feature is also one that has been neglected by current measurement practices (e.g. Langan-Fox, Code, & Langfield-Smith, 2000). There has been significant theoretical work delineating cognitive constructs such as team decision making, shared mental models, and team situation awareness (CannonBowers, et al., 1993; Orasanu, 1990; Stout, Cannon-Bowers, & Salas, 1996). It is assumed that with an understanding of these constructs, training and design interventions can target the cognitive underpinnings of team performance. Also, the hypothesized relation between team cognition and team performance suggests that team performance can be predicted from an assessment of team cognition. The ability to predict team performance from team cognition suggests that team performance may ultimately be assessed indirectly through cognitive measures independent of the task, thereby circumventing the need for teams to perform in less than optimal settings (e.g., hazardous or high-risk environments). The research presented in this article focuses on team knowledge. Parallel to research on individual expertise (e.g., Chase & Simon, 1973; Glaser & Chi, 1988), accounts of effective team performance highlight the importance of knowledge, or in this case, team knowledge. For instance, Cannon-Bowers and Salas (1997) have recently proposed a framework that integrates many aspects of team cognition in the form of teamwork competencies. They categorize competencies required for effective teamwork in terms of knowledge, skills, and attitudes that are either specific or generic to the task and specific or generic to the team. Similarly, a team's understanding of a complex and dynamic situation at any one point in time (i.e., team situation awareness) is supposedly Measuring Team Knowledge 6 influenced by the knowledge that the team possesses (Cooke, Stout, & Salas, 1997; Stout, et al., 1996). Figure 1 presents team knowledge in a framework that serves to better define team knowledge, and at the same time to identify some issues in its measurement. Traditional measures of team knowledge operate at the collective level, eliciting knowledge from individuals on the team and then aggregating the individual results to generate a representation of the collective knowledge of a team. Although collective knowledge should be predictive of team performance, it is devoid of the influences of team process behaviors (e.g., communication, coordination, situation awareness), analogous to individual cognitive processes. The process behaviors transform the collective knowledge into effective knowledge. This effective knowledge is what we describe as the holistic level and is associated with actions and ultimately, with team performance. Also, note in Figure 1 that team knowledge consists of background knowledge that is long-lived in nature, as well as more dynamic and fleeting understanding that an operator has of a situation at any one point in time. Measures of team knowledge have focused primarily on the former, at the expense of the latter. [Insert Figure 1 about here] Other measurement issues include the fact that traditional measures of team knowledge, focusing primarily on the similarity of team members’ knowledge, tend to assume homogeneity with respect to team composition as opposed to the heterogeneous backgrounds suggested by the definition of team offered above. In addition, methods for Measuring Team Knowledge 7 aggregating individual knowledge to derive collective knowledge are worthy of further study (e.g., the social decision scheme literature (Hinz, 1999; Davis, 1973)), as are measures that target different types of team knowledge (e.g., strategic, declarative, procedural knowledge or task vs. team knowledge). Finally, measures of team knowledge could benefit from a broader application of various knowledge elicitation techniques, procedures to better automate the measures and embed them within task contexts, and evaluations of measures in terms of validity and reliability. Details of these issues are described in Cooke, et al. (2000). The team knowledge measures that are evaluated in this paper attempt to address some of these measurement issues. In particular, measures target both long-term background knowledge of both taskwork and teamwork varieties, as well as knowledge associated with the more dynamic situation models (i.e., a team’s understanding of the situation at one point in time). In addition, one of the elicitation methods targets the holistic level and a set of team knowledge metrics is used to take into account the heterogeneous nature of the teams. The Synthetic Task Environment We assume that the task and surrounding environment are inextricably tied to team knowledge and its measurement. This makes the selection of the team task and setting critical. While field settings provide realistic opportunities for observation, they do not afford the experimental control and measurement flexibility needed for the development and evaluation of measures. Therefore, these studies were conducted in the context of an STE (Synthetic Task Environment), based on the real task of controlling a UAV (Uninhabited Air Vehicle). Measuring Team Knowledge 8 Synthetic tasks are "research tasks constructed by systematic abstraction from a corresponding real-world task" (Martin, Lyon, & Schreiber, 1998, p. 123). An STE provides the context for a suite of synthetic tasks. Performance on a synthetic task should exercise some of the same behavioral and cognitive skills associated with the real-world task. This environment offers a research platform that bridges the gap between controlled studies using artificial laboratory tasks and uncontrolled field studies on real tasks or using high-fidelity simulators. The STE used in these studies is an abstraction of the Air Force's Predator UAV operations (Cooke, Rivera, Shope & Caukwell, 1999; Cooke, Shope, & Rivera, 2000). It is implemented in the context of NMSU’s CERTT (Cognitive Engineering Research on Team Tasks) Laboratory that contains hardware and software for simulating a variety of team tasks and adequate measurement and task manipulation capabilities for research on those tasks. CERTT's UAV-STE is a three-person task in which each team member is provided with distinct, though overlapping, training; has unique, yet interdependent roles; and is presented with different and overlapping information during the mission. The overall goal is to fly the UAV to designated target areas and to take acceptable photos at these areas. The AVO (Air Vehicle Operator) controls airspeed, heading, and altitude, and monitors UAV systems. The PLO (Payload Operator) adjusts camera settings, takes photos, and monitors the camera equipment. The DEMPC (Data Exploitation, Mission Planning and Communication Operator) oversees the mission and determines flight paths under various constraints. To complete the mission, the team members need to share Measuring Team Knowledge 9 information with one another and work in a coordinated fashion. Most communication is done via microphones and headsets, although some involves computer messaging. The CERTT UAV-STE was abstracted from results of a cognitive task analysis (Gugerty, DeBoom, Walker, & Burns, 1999) of the Predator operational environment, with the goal of providing an experimenter-friendly test-bed for the study of team cognition. As a result, cognitive aspects of the task are emphasized and other task components (e.g.. the specific interface, stick-and-rudder control have been omitted). For instance, alterations in the interface enable individual team members to rapidly (within 1.5 hours) acquire the skills and knowledge needed to work as an integral part of the team. Measures taken include audio records, video records, digital information flow data, embedded performance measures, team process behavior measures, and a variety of individual and team knowledge measures. Overview of Study The study was conducted in the context of the team UAV task. Eleven teams of three Air Force ROTC cadets participated in three experimental sessions lasting from three to six hours. During these sessions teams were trained on the task and were observed as they performed ten 40-minute missions. During each mission team performance, team process behavior and team situation models were measured. In addition, long-term team knowledge regarding both taskwork and teamwork were measured apart from the missions in four sessions. This study was designed to evaluate a number of different approaches to measuring team knowledge and to examine the development of team performance, process, and knowledge as team skill was acquired over the ten missions. The validity of team knowledge measures was assessed in terms of Measuring Team Knowledge 10 the ability of measures to predict team performance and process. In addition the patterns of acquisition of team knowledge, performance, and process were also examined. These patterns may shed light on sequential dependencies among components of team performance and in addition, provide useful information about the point at which asymptotic performance is reached in this synthetic task METHOD Participants Eleven three-person teams of Air Force ROTC cadets voluntarily participated in three (3-5 hour) sessions of this study. Individuals were compensated for their participation by payment of $6.00 per person hour to the ROTC organization. In addition, the three team members on the team of with the highest mean performance score were each awarded a $50.00 bonus. Equipment and Materials The study took place in New Mexico State University’s CERTT (Cognitive Engineering Research on Team Tasks) Lab, configured for the UAV team synthetic task described above. For most of the study, each participant was seated at a workstation consisting of two computer monitors (one View Sonic monitor connected to an IBM PC 300PL and one Cyberesearch Industrial Workstation), a Sony video monitor that could present video from a Quasar VCR, a keyboard, a keypad, and a mouse for input. Participants communicated with each other and the experimenters using David Clark headsets and a custom-built intercom system, designed to log speaker identity and time information. The intercom enabled participants to select one or more listeners by flipping toggle switches. Measuring Team Knowledge 11 Two experimenters were seated in a separate adjoining room at an experimenter control station consisting of another IPB PC computer and View Sonic monitor, headsets for communicating with participants, and Panasonic monitors for video feed from ceilingmounted Toshiba cameras located behind each participant. In addition, a fourth camera captured information from the entire participant room. From the experimenter workstation, the experimenters could start and stop the mission, query participants together or individually, monitor some of the mission-relevant displays, observe team behavior through camera and audio input, and enter time-stamped observations. Video data from cameras were recorded on a Quasar VCR. Audio data from the headsets were recorded on an Alesis digital recorder as well as to the VCR. In addition, custom software recorded communication events in terms of speaker, listener, and the interval in which the push-to-talk button on the microphone was depressed. Custom software (seven applications connected over a local area net) also ran the synthetic task and collected values of various parameters that were used as input by performance scoring software. A series of tutorials were designed in Powerpoint for training the three team members. Two of the three Powerpoint modules were unique to each position. Custom software was also developed to conduct tests on information in Powerpoint tutorials, to collect individual and consensus taskwork relatedness ratings, and to collect demographic and preference data at the time of debriefing. In addition to software, some mission-support materials (rules-at-a-glace for each position, two screen shots per station corresponding to that station’s computer displays, and examples of good and bad photos for the Payload Operator) were presented on paper at the appropriate workstations. Other paper materials consisted of the consent forms, Measuring Team Knowledge 12 debriefing form, a checklist of skills for training, forms for experimenter recording of responses to situation model queries and observations of process behaviors, a trust survey, and teamwork and taskwork questionnaires. Measures Performance, process, and knowledge measures are the focus of this paper, though demographic, preference, trust, video, and communication data were also collected; they are not addressed in this article. Team performance was measured using a composite score based on the result of mission variables including time each individual spent in an alarm state, amount of fuel used, amount of film used, number of targets successfully photographed, and number of critical waypoints visited. Penalty points for each of these components were weighted a priori in accord with importance to the task and subtracted from a maximum score of 1000. Team process behavior was scored independently by each of the two experimenters. For each mission the experimenters observed team behavior and responded yes or no to each of nine team process behaviors based on whether that behavior did or did occur at designated event-triggers in each mission. Team process was simply the proportion of the nine process questions that were observed by each experimenter. The process behaviors and triggers are presented in Table 1. Similar questions were used for Missions 7 and 10, with different event details to accommodate the different scenario. [Insert Table 1 about here] Measuring Team Knowledge 13 Team knowledge was measured using several different methods outlined in Table 2. Team situation models were measured using three SPAM-like (Durso, Hackworth, Truitt, Crutchfield, Nikolic, & Manning, 1998) queries administered during the mission. Query order and the time (in 5 minute increments) at which queries began were both randomly determined without replacement. One of the experimenters administered the queries to each team member in turn during the five-minute interval. Order in which team members were queried was also random. The three queries asked (1) a prediction regarding the number of targets out of nine successfully photographed by the end of the mission; (2) the team member or members that they would communicate with next and the topic of that communication; and (3) the number of targets out of nine successfully photographed thus far. The experimenter also recorded the correct response to these queries once known. Responses to the queries were scored for accuracy, as well as intrateam similarity. Team accuracy scores were based on the average accuracy of team members, as scored using the experimenter-generated key. For the second query, this was simply the proportion of correct responses (1 or 0) averaged across the three team members. For the first and third queries, this was the absolute value of the deviation from correct, divided by 9 possible targets and subtracted from 1. For the first and last queries, team similarity was the average of all the pairwise similarities (i.e., converse proportions of absolute deviations) of the three team members. Intrateam similarity was not meaningful for the second query. [Insert Table 2 about here] Measuring Team Knowledge 14 Longer-term team knowledge was measured in four separate sessions by four methods: teamwork questionnaire, taskwork questionnaire, taskwork ratings, and taskwork consensus ratings. The teamwork questionnaire consisted of a three-part question in which the individual was asked to indicate if directed pairs of team members (e.g., AVO PLO) pass information in the specified direction. The second part of the question asked them to identify the nature of the information for those communication links identified. The third part asked them to consider any sequential constraints in the timing of the information. The taskwork questionnaire asked individuals to analyze the task starting with the main goal and breaking this up into subgoals and tasks. The next part of this questionnaire asked individuals to associate team roles with each of the tasks and then to indicate any sequential constraints in tasks. The taskwork ratings consisted of eleven task related terms: altitude, focus, zoom, effective radius, ROZ entry, target, airspeed, shutter speed, fuel, mission time, and photos. All possible pairs of these terms were presented in one direction only, one pair at a time. Pair order was randomized and order within pairs was counterbalanced across participants. Each team member rated the relatedness of each pair on a 1-5 scale with anchors that ranged from slightly related to highly related. There was also an option of unrelated. Taskwork consensus ratings consisted of the same pairs as taskwork ratings (randomly presented), however the ratings were entered as a team. For each pair, the rating entered in the prior session by each team member was displayed on the computer Measuring Team Knowledge 15 screen of that team member. The three team members discussed each pair over their headsets until consensus was reached. The longerterm knowledge measures were each scored for accuracy and intrateam similarity. Individual accuracy scores and pairwise measures of response similarity were averaged across team members. For the two rating tasks, data were submitted to KNOT (using parameters r=inf. And q=n-1) in order to generate Pathfinder networks (Schvaneveldt, 1990). These networks reduce and represent the rating data in a meaningful way in terms of a graph structure with concept nodes standing for terms and links standing for associations between terms. A referent network generated by the experimenters served as the key, and similarity of any one network to this referent in terms of the proportion of shared links was used as a measure of accuracy. In addition, the individual task ratings were scored not only against a key representing overall knowledge, but also against role-specific keys. In this way, measures of “role” or “positional” accuracy, as well as “interpositional” accuracy (i.e., interpositional knowledge (IPK) or knowledge of roles other than their own) could be determined. Team accuracy was the mean accuracy across team members. Intrateam similarity was measured using the proportion of shared links for all intrateam pairs of two individual networks (i.e. the mean of the three pairwise similarity values among the three