Modeling Proficiency in a Tailored, Situated Training Environment

Situated training via simulation has many beneficial properties including the potential to adapt or to tailor training experience to the evolving needs and competencies of individual learners. Such tailoring often requires estimates of learner proficiency. These estimates inform the tailoring that is undertaken by the system. In this paper, we propose that proficiency models must be analytic, informal, and transparent to support situated training systems. We then assess the extent to which some previous approaches to proficiency modeling meet these requirements and describe several enhancements to an established proficiency-modeling paradigm. We hypothesize these enhancements will enable practical use of proficiency modeling in situated training systems. We illustrate with examples from application in a perceptual training system currently under development. 1. Tailored, Situated Training Situated training refers to demonstration, practice, and assessment of skills in a representative environment, often a simulated one, with scenarios or background stories to link the skills. Situated training gives learners context for the skills being trained, supporting their ability to recognize cues and construct mental models for increased acquisition speed, retention, and transfer (Oser, Cannon-Bowers, Salas, & Dwyer, 1999). Tailored training adds individualization of scenario experience, so that scenarios become more closely aligned with the needs of individual learners in terms of progression, support, and instructional strategy selection (Wray & Woods, 2013). In a situated training environment, tailoring can change scenario events and story details to better meet training needs. These simulation-intrinsic interventions, as opposed to extrinsic interventions such as text boxes or explicit tutor dialogs, introduce an ability to assess and remediate target skills within the context of a situation without distracting the learner from the simulated experience or decreasing learner engagement. To deliver such an individualized experience, a model of the student’s proficiency is typically needed The information stored in our proficiency model includes a list of domain skills, estimates of individual proficiency for each skill, and descriptions of how the estimates vary in reaction to specific observations about the learner and changes elsewhere in the model (via skill relationships). The proficiency estimates are then queried to decide (along with other factors not covered here) what tailoring is appropriate. In addition to modeling proficiency itself, situated training and practical constraints impose additional requirements on the design of the proficiency model component. For example, the model we describe is not a strongly principled cognitive model in comparison to, e.g., Anderson and colleagues many years of work carefully modeling skill acquisition (e.g., Anderson, 1982). Instead, we are targeting an operational mechanism that enables different modes of adaptation. The model may not predict skill level as accurately or precisely as other methods. However, we see one of the advantages of the "fuzzy" categorization we use within the model is that the estimate of the learner's skill is not tied to a precise and fixed assessment. This paper describes these requirements and then outlines a fuzzy vector proficiency model (Katz & Lesgold, 1991; Lesgold, Lajoie, Bunzo, & Eggan, 1988), enhanced with new capabilities targeting improved handling of situated data. Although the proficiency modeling approach is designed for use across multiple domains, we focus here on how it facilitates the delivery of tailored, situated training for a specific advanced military skills training simulator. 1.1 Requirements for proficiency modeling The Virtual Observation Platform (VOP) is an immersive simulation that trains perceptual and cultural skills in the setting of a military observation post and its environs (Schatz, Wray, Folsom-Kovarik, & Nicholson, 2012). Skill practice is situated in the context of an ongoing mission and framing story, and the details of the scenario can be tailored in real time through the VOP’s Dynamic Tailoring System (DTS). In building this complex system (and others) for realworld military training, we are identifying requirements for a trainee proficiency model. The DTS carries out tailoring that is designed to manipulate the pedagogical experience through changes intrinsic to the scenario. Tailoring is used to scaffold (Pea 2004) and fade scaffolding based on estimated learner ability, as well as to increase challenge for the purpose of motivating learning (Vygotsky, 1978). For example, in the VOP, if a novice trainee is believed to have low proficiency in the perceptual skill of noticing human kinesic (gestural) cues then the DTS might exaggerate the magnitude and visual salience of a simulated character’s movement to support the trainee seeing that gesture. Conversely, as the trainee becomes more expert in recognizing gestures the DTS might introduce visual distractors to challenge performance. Tailoring in the VOP can differentially affect a range of separate skills (Fautua et al., 2010) that combine to affect observable trainee performance simultaneously. For example, a scenario in the VOP might require correct visual scan technique for the trainee to see an event taking place, sociocultural sensemaking skills (Klein, Moon, & Hoffman, 2006) to correctly interpret the importance of the observed event, and finally correct selection and execution of doctrinal responses to the event such as reporting an important observation to teammates or commanders. Therefore, proficiency estimates for multiple interrelated skills are needed to effectively tailor situated training of different skills. Because training is situated, the interpretation of performance is more complex. Interpretation must take place in the context of previous proficiency estimates, variable task complexity and difficulty, current level of support or challenge that the tailoring system has implemented, and likely slips or guesses that we expect might make a particular skill performance unreliable in indicating actual underlying ability. Further, instructor judgment is a valuable source of proficiency input. While technically challenging to collect in a userfriendly and useful manner, instructor input must be incorporated in the DTS both to improve interpretation and to gain user acceptance. An important contribution of this work is the ability for a proficiency model to account for all these sources of context. Mitigating the complexity of the requirements outlined thus far, there is a useful commonality shared by the kinds of intrinsic pedagogical tailoring that the DTS executes. While intrinsic tailoring does create a meaningful scenario difference between what experiences are appropriate at each proficiency level, it also provides some forgiveness if the estimated level differs from a trainee’s actual proficiency. Unlike simulation-extrinsic pedagogical interventions (e.g., initiating a dialog to correct an error), intrinsic interventions are unlikely to interfere with learning or even distract the trainee if they are occasionally displayed incorrectly. If the DTS estimates a trainee has not observed a target behavior when he actually has, then increasing the behavior’s salience has less negative impact on training efficacy than displaying an error message to the trainee. The low commitment of intrinsic tailoring to a particular skill estimate is valuable to the design we present. Based on the requirements associated with situated training, we next briefly explore differences between several candidate models of trainee proficiency. 1.2 Comparing proficiency models for tailored,