Measuring Team Knowledge 1 Running Head: MEASURING TEAM KNOWLEDGE Measuring Team Knowledge: A Window to the Cognitive Underpinnings of Team Performance

This paper reports an effort aimed at developing and evaluating measures of taskwork and teamwork team knowledge for teams in which members differ in knowledge backgrounds. These measures are used in a study with 36 teams to explore the cognitive underpinnings of team performance variations due to cross training regime. We demonstrate that these measures are valid and provide team performance information that complements outcome and behavioral measures. Teams exposed to full cross training acquired more taskwork and teamwork knowledge than control teams or teams exposed to a conceptual version of cross training. Measures of team knowledge provide information regarding team task performance critical for system design and training programs. Measuring Team Knowledge 3 Measuring Team Knowledge: A Window to the Cognitive Underpinnings of Team Performance Team process behaviors such as communication, leadership behaviors, coordination, and planning have been linked theoretically and empirically to team performance (Foushee, 1984; Stout, Salas, & Carson, 1994; Zalesny, Salas, & Prince, 1995). Many interventions for improving team performance have targeted team process behavior (Braun, Bowers, Holmes, & Salas, 1993; Leedom & Simon, 1995; Prince, Chidester, Cannon-Bowers, & Bowers, 1992; Prince & Salas, 1993). Recently it has become clear that other factors that are more cognitive than behavioral in nature also play a role in team performance. An overall objective of the work presented here is to develop valid cognitive measures for teams. Technological developments in the military and elsewhere have transformed highly repetitive manual tasks, requiring practiced motor skills, to tasks that require cognitive skills often related to overseeing new technology such as monitoring, planning, decision making, and design (Howell & Cooke, 1989). As a result, a full understanding of many tasks, at a level required to intervene via training or system design, requires an examination of their cognitive underpinnings. Additionally, the growing complexity of tasks frequently surpasses the cognitive capabilities of individuals and thus, necessitates a team approach, which simultaneously introduces an additional layer of cognitive requirements that are associated with the demands of working together effectively with others. Team members need to coordinate their activities with others who are working toward the same goal. Team tasks often call for the team to detect and recognize pertinent cues, make decisions, solve problems, remember relevant information, plan, Measuring Team Knowledge 4 acquire knowledge, and design solutions or products as an integrated unit. Therefore, an understanding of team cognition, or what some have called the new "social cognition" (Klimoski & Mohammed, 1994; Larson & Christensen, 1993; Nye & Brower, 1996), is critical to understanding team performance and intervening to prevent errors or improve productivity and effectiveness. In this paper we intentionally restrict our focus to team knowledge. According to Salas, Dickinson, Converse, and Tannenbaum (1992), a team is "a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership" (p. 4). Thus, based on this definition, a team is a special type of group (Hackman & Walton, 1986). Although there has been significant work on group cognition (e.g., Davis, Au, Hulbert, Chen, & Zarnoth, 1997; Hinsz, Tindale, & Vollrath, 1997; Stasser, Stewart, & Wittenbaum, 1995; Steiner, 1972; Thompson, Levine, & Messick, 1999; Wegner, 1986), our focus on teams as a type of group, presents special challenges for the measurement of team cognition. Specifically, the fact that team members are assigned distinct, though interdependent roles, raises issues regarding the concept of shared knowledge. The methods that we discuss in this paper address these and other issues. There has also been significant theoretical work delineating cognitive constructs at the team level such as shared mental models and team situation awareness (CannonBowers, Salas, & Converse 1993; Orasanu, 1990; Stout, Cannon-Bowers, & Salas, 1996) for which team knowledge is thought to be central (Cooke, Salas, Cannon-Bowers, & Stout, 2000). It is assumed that understanding these constructs will allow diagnosis of Measuring Team Knowledge 5 team performance, which is useful for training and design interventions. Also, the hypothesized relation between team cognition and team performance suggests that team performance can be predicted from an assessment of team cognition and perhaps apart from the performance context, thereby providing an alternative to assessment requiring teams to perform in suboptimal settings (e.g., with minimal training, in hazardous or high-risk environments). Team knowledge is a component of team cognition that includes constructs such as shared mental models and team situation models. Parallel to research on individual expertise (e.g., Chase & Simon, 1973; Glaser & Chi, 1988), accounts of effective team performance highlight the importance of knowledge, or in this case, team knowledge. For instance, Cannon-Bowers and Salas (1997) have recently proposed a framework that integrates many aspects of team cognition in the form of teamwork competencies. They categorize competencies required for effective teamwork in terms of knowledge, skills, and attitudes that are either specific or generic to the task and specific or generic to the team (see also Stevens and Campion, 1994; 1999). Their distinction between teamwork and taskwork knowledge builds on the distinction made by Morgan, Glickman, Woodard, Blaiwes, and Salas (1986). The important role of team knowledge has also been empirically supported in several studies examining shared mental models and their relation to team performance (e.g., Marks, Zaccaro, & Mathieu, 2000; Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000). In general, team members with mental models that are accurate and more similar to one another tend to perform at higher levels compared to team members with dissimilar and inaccurate models. Another example, in which team knowledge plays a critical role in team performance, is team situation Measuring Team Knowledge 6 awareness or the team’s understanding of a complex and dynamic situation at any one point in time. The team’s ability to assess the situation is supposedly influenced by the fleeting knowledge of the situation that the team possesses or a “team situation model” (Cooke, Stout, & Salas, 1997; Stout, et al., 1996). Thus, like team cognition, we assume that the more specific measurement of team knowledge can enhance our understanding of team performance and the factors affecting it and provide diagnostic information for team training and design. The measurement of team knowledge, however, is replete with questions and unresolved issues (Cooke, et al., 2000). For instance, Mohammed, Klimowski, & Rentsch (2000) note that there are a number of methods for measuring team mental models, each suited to different purposes. Further, in the few cases in which team knowledge has been measured, that measurement has focused primarily on team member similarity and to a lesser extent, overall accuracy (e.g., Langan-Fox, Code, & LangfieldSmith, 2000). This focus seems suboptimal for teams in which individuals have distinct, yet interdependent roles (Salas, Dickinson, Converse, & Tannenbaum, 1992) and thus, may share knowledge in the sense that it is distributed among, rather than similar across team members. Furthermore, measures have failed to distinguish taskwork and teamwork knowledge. Other limitations of team knowledge measurement are reviewed in Cooke, et al. (2000). In short, the reliable and valid measurement of constructs like team knowledge is a first, albeit nontrivial step toward advancing our understanding of team cognition. Therefore, the objectives of this research were to 1) develop and evaluate measures of team knowledge relevant to taskwork and teamwork and suitable for teams in which Measuring Team Knowledge 7 knowledge is distributed across team members (i.e., they have heterogeneous backgrounds) and 2) to use these measures to better understand the cognitive underpinnings of team performance variations due to training strategy differences. Effective measures of team knowledge should correspond to performance differences among teams, but should also reveal knowledge differences that offer explanations for the success or failure of various training strategies. Cross training was the specific training strategy investigated in this experiment and is defined by Volpe, CannonBowers, Salas, and Spector (1996) as a “strategy in which each team member is trained on the tasks, duties, and responsibilities of his or her fellow team members” (pp. 87). Cross training was selected because it has resulted in performance benefits in laboratory studies and these benefits are purportedly tied to the acquisition of taskwork and teamwork knowledge among members of heterogeneous teams. Cross Training Empirical evidence exists to support the effectiveness of cross training on team performance (Baker, 1991; Cannon-Bowers, Salas, Blickensderfer, & Bowers, 1998). Cross training has been thought to be effective because it promotes the shift from heterogeneous team members (i.e., who specialize in their own roles) to homogeneous members who understand the other roles as well. That is, there is an assumed shift from less to more IPK (interpositional knowledge). IPK is knowledge relevant to team positions other than one’s own position. However, the state-of-the-art in team knowledge measurement has precluded direct evidence for these types of team knowledge changes with cross training. The single exception is work by Cannon-Bowers, et al. (1998). In this study three-person teams were cross-trained or not in the other positions for a Measuring Team Knowledge 8 command and control task. As a manipulation check, individuals completed a 33-item matching test that requested the type of information needed by the various positions to make decisions (i.e., IPK). Cross-trained teams exhibited higher levels of IPK than teams that were not cross trained. The type of cross training that has been empirically demonstrated to be effective includes a substantial portion of hands-on practice on the tasks of other team members. Most commonly, team members are trained on the other positions to the same extent that they are trained on their own positions. Whereas, this kind of hands-on-cross-training is possible in the context of scaled tasks associated with simulations and laboratory experiments, it becomes expensive and highly time consuming as task complexity increases and approaches the complexity inherent in most “real world” tasks (e.g., consider cross training surgeon and nurse positions). For this reason, cross training is not widely used by most organizations. Additionally, while cross training may benefit individual performance through insights gained from different perspectives, CannonBowers, et al. (1998) note that training on multiple complex and distinct positions may increase the possibility of individual proficiency decrements. In other words, assuming a limited capacity for skill acquisition, specialization and skill on the individual's own job is traded off for a broader range of skills associated with all jobs. Blickensderfer, Stout, Cannon-Bowers, and Salas (1993) suggest that the same benefits of full cross training may be realized in training that is less intense, though focused on achieving an understanding of the positions and their interdependencies. They recommend using shared mental model theory to drive an abbreviated form of cross training with a focus on what actions other team members perform, as opposed to why or Measuring Team Knowledge 9 how they are performed. In a sense, this is more of a teamwork orientation, as opposed to the taskwork orientation of traditional cross training. Indeed the study by Volpe, et al. (1996) demonstrated that a 10 minute intervention targeting the roles and responsibilities of the other position on a two-person team resulted in significant performance benefits over no cross training. Therefore, in this study, we compare a conceptual, abbreviated form of crosstraining to full cross training. In full cross training team members proceed through the full-training program in each team position. The conceptual cross training is based on a shared mental models perspective and specifically targets the acquisition of teamwork IPK. It is hoped that conceptual training can achieve some or all of the same benefits of full cross training, while minimizing expense, training time, and possibilities of individual proficiency decrements. At the same time, the application of knowledge measures should provide a deeper look at the effects of these different training interventions on team knowledge. Study Overview and Hypotheses In this study four training conditions were compared: 1) FCT (full cross training), 2) CCT-35 (conceptual cross training for 35 minutes), 3) CCT-75 (conceptual cross training for 75 minutes), and 4) Control. The FCT condition required 75 minutes of training time, whereas the CCT condition by definition required less training time (i.e., 35 minutes). Therefore, in order to control for training time, participants in the CCT-75 condition spent time on the conceptual cross training material equivalent to time spent by FCT participants. In the Control condition, participants were trained only on their own role also over a 75-minute period to again control for time differences. Team Measuring Team Knowledge 10 performance and team knowledge (taskwork and teamwork) were measured. Different team knowledge metrics were developed to distinguish overall, positional, and IPK knowledge accuracy. It was first hypothesized that to the extent that our taskwork and teamwork measures were valid, that they should predict team performance differences. We therefore, predict that teamwork knowledge accuracy and similarity indices should be positively correlated with team performance (Hypothesis 1). Further, to the extent that interpositional knowledge is important for effective performance on this task, this should be reflected in relatively strong correlations between IPK accuracy and team performance (Hypothesis 2). From previous studies, we predict a benefit of cross training of any type over the control condition. We further predict that CCT-35 should have performance benefits over the Control condition, yet comparable to those of FCT, while requiring less training time (Hypothesis 3). Differences between CCT-35 and FCT conditions and the CCT-75 condition should indicate the degree to which training time as opposed to training content are responsible for any benefits of FCT over CCT-35. Performance differences between training conditions should also be reflected in the knowledge measures. Interpositional knowledge accuracy should be greater for the cross-trained teams over the control teams (Hypothesis 4). Specifically, both the traditional and conceptual varieties of cross training should result in superior IPK over the control condition, however, because of the different foci, the former should result in superior taskwork knowledge, and the latter in superior teamwork knowledge (Hypothesis 5). Measuring Team Knowledge 11