Running head: DUAL LEARNING PROCESSES IN INTERACTIVE SKILL ACQUISITION Dual Learning Processes in Interactive Skill Acquisition
Abstract
Acquisition of interactive skills involves the use of internal and external cues. Experiment 1 showed that when actions were interdependent, learning was effective with and without external cues in the single-task condition but was effective only with the presence of external cues in the dual-task condition. In the dual-task condition, actions closer to the feedback were learned faster than actions farther away but this difference was reversed in the single-task condition. Experiment 2 tested how knowledge acquired in single and dual-task conditions would transfer to a new reward structure. Results confirmed the two forms of learning mediated by the secondary task: A declarative memory encoding process that simultaneously assigned credits to actions and a reinforcement-learning process that slowly propagated credits backwards from the feedback. The results showed that both forms of learning were engaged during training, but only at the response selection stage, one form of knowledge may dominate over the other depending on the availability of attentional resources. Dual Learning Processes in Interactive Skill Acquisition The acquisition of interactive skills often involves the retention and utilization of internal cues (e.g., episodic memory of previous actions) and the recognition of relevant external cues (e.g., signs on the walls). The utilization of both internal and external cues to inform actions can be commonly found in most human-technology interactions (e.g., Card, Moran, & Newell, 1983; Fu, & Pirolli, 2007), in which the operators have to encode and recall previous actions and their outcomes and to interpret ongoing information displayed on an interface to decide what to do next. A better understanding of the interactive processes involved in the use of these internal and external cues is critical in predicting learning and performance in different designs of humantechnology interfaces. Understanding interactive skills requires the careful study of how internal and external cues are processed during the interplay of cognition and perception (Ballard, Hayhoe, Pook, & Rao, 1997; Fu & Anderson, in press; Fu & Gray, 2000, 2004, 2006; Gray & Fu, 2004; Gray, Sims, Fu, & Schoelles, 2006; Larkin, 1989). Our focus is on one of the most typical situations in which immediate feedback on individual actions is not available, so that the learner has to perform a sequence of actions and gets feedback on their success at the end of the sequence. This kind of situation creates a difficult credit-assignment problem, in which the learner has to assign credits to earlier actions or external cues that are responsible for eventual success. The creditassignment problem is even more difficult in dynamic environments in which the action outcomes are probabilistic and interdependent, and the environment may change both autonomously and as a result of the actions. In this case, either memory of previous actions or recognition of the correct cues in the external environment is required to properly assign credits to the appropriate actions. Based on the recent findings from psychological research, we hypothesize that humans exhibit two distinct learning processes to solve this credit assignment problem: a declarative process that encodes internal memory cues for actions and outcomes, and a reinforcementlearning process that encodes the relationship between the external cues and action outcomes (e.g., Daw, Niv, & Dayan, 2005; Grafton, Hazeltine, & Ivry, 1995; Keele, Ivry, Mayr, Hazeltine, & Heuer, 2003; Knowlton, Squire, & Gluck, 1994; Knowlton, Mangels, & Squire, 1996; Packard, & Knowlton, 2002; Poldrack, Clark, Pare-Blagoev, Shohamy, Moyano, Myers, & Gluck, 2001; Squire, 1992; Waldron, & Ashby, 2001). One major source of evidence for this distinction is from studies that show that amnesic patients can perform some tasks (e.g., artificial grammar, sequence, or category learning) despite their lack of declarative access to the events of training (e.g., Foerde, Knowlton, & Poldrack, 2006; Knowlton et al., 1994; Nissen, & Bullemer, 1987), whereas patients with basal ganglia disorders show major impairment to similar tasks (Poldrack et al., 2001). Furthermore, recent findings from neuroscience show that neural activities in the basal ganglia correlate well with the predictions of reinforcement learning when learning occurs in various probabilistic reward structures (e.g., Schultz, Dayan, & Montague, 1997). These two sets of results suggest that there exist two distinct learning systems for probabilistic events. In this article, we describe results from two experiments designed to tease apart the nature of these two learning systems in the acquisition of interactive skills. We define interactive skills broadly in terms of learning of action sequences in situations that depend critically on the utilization of external cues. Our experiments are designed by bridging together two lines of psychological research: First, learning the sequential nature of actions is related to the research on sequence learning; second, learning the probabilistic relationship among external cues, previous actions, and their outcomes is related to the research on probability learning and probabilistic classification. Sequence Learning The dual learning processes have often been investigated through a paradigm called sequence learning (e.g., Cohen, Ivry, & Keele, 1990; Curran, & Keele, 1993; Nissen, & Bullemer, 1987; Willingham, Nissen, & Bullemer, 1989). One typical paradigm is the serial reaction time (SRT) task in which participants have to press a sequence of keys as indicated by a sequence of lights. A certain pattern of button presses recurs regularly and participants give evidence of learning this sequence by pressing the keys for this sequence faster than a random sequence. One common finding is that learning is observed as a facilitation of test performance without concomitant awareness of what is being learned. A number of studies have used a secondary task such as counting of tones to study the effects of diminished attention for sequence learning (e.g., Cohen et al., 1990; Curran, & Keele, 1993; Nissen, & Bullemer, 1987). Cohen et al. (1990) found that when attention is diminished by a secondary task, participants could only learn simple pairwise transitions, but failed to learn higher order hierarchical structures in the sequence. The results show that although sequence learning can occur with diminished attention, its scope is limited to simple cases in which simple associations of stimuli are sufficient to perform the task. Probability Learning and Probabilistic Classification In a typical probability-learning experiment, participants guess which of the alternatives occurs and then receives feedback on their guesses (e.g., Estes, 1964). One robust finding is that participants often “probability match”; that is, they will choose a particular alternative with the same probability that it is reinforced (e.g., Friedman, Burke, Cole, Keller. Millward, & Estes, 1964). This leads many to propose that probability matching is the result of a habit-learning mechanism that accumulates information about the probabilistic structure of the environment (e.g., Graybiel, 1995; Knowlton et al., 1994). One important characteristic of this kind of habit learning is that information is acquired gradually across many trials, and appears to be independent of declarative memory as amnesic patients could learn to perform in a probabilistic classification task (Knowlton et al., 1994; but see Gallistel, 2005). However, for non-amnesic human participants, it is difficult to determine whether this kind of probabilistic classification is independent of the use of declarative memory. Given that declarative memory is dominant in humans, it has been argued that learners often initially engage in declarative memory encoding in which they seek to remember sequential patterns even when there are none (Yellott, 1969). Researchers argue that true probabilistic trial-by-trial behavior only appears after hundreds of trials – perhaps by then participants give up the idea of explicitly encoding patterns and the habit-learning process becomes dominant (Estes, 2002; Vulkan, 2000). Similarly, recent research on complex category learning has also provided interesting results suggesting the dual learning systems (Allen, & Brooks, 1991; Ashby, Queller, & Berretty, 1999; Waldron, & Ashby, 2001). Present Approach In both the sequence-learning and the probability-learning paradigms, participants do not need to learn from the delayed feedback of a single action as immediate feedback is given. In a typical SRT task there is a sequence of actions but there is a deterministic relationship (given by instructions) between the stimuli and their responses. Participants in the SRT may anticipate the next stimuli but they always get immediate feedback after their responses. In probability learning the stimulus-response relationship is probabilistic but there is a single action after which feedback is received. Neither of these paradigms then directly reflects the complexity of the credit-assignment problem in interactive tasks in which people often have to learn to sequentially choose actions with probabilistic outcomes and receive feedback only after the whole action sequence is executed. Our studies were designed by combining research from both areas by studying the nature of the learning processes that assign credits to different actions and external cues in a probabilistic sequential choice task. In this task, a sequence of actions was executed before feedback on its correctness was received, and a particular action sequence was correct only with a certain probability. Our goal is to use the novel paradigm to investigate the nature of the dual learning processes in the general context of interactive skill learning when the learner has to choose the right action sequences by utilizing either memory or external cues in the environment. The Probabilistic Sequential Choice Task A probabilistic sequential choice task was designed in which we predicted different behavioral patterns exhibited by the declarative memory encoding and reinforcement learning processes. At the outset, we would like to stress that although previous studies on the dual learning processes aimed at identifying distinct neural substrates for these two learning processes, our goals were different in that we aimed at finding out the differences in the nature of the learning process that is associated with declarative knowledge of task (declarative memory encoding) and the learning process that is associated with tacit knowledge that cannot be verbalized (reinforcement-learning process). In addition, we aimed at finding out how these two learning processes may predict different behavioral outcomes and how they may interact in different situations during the acquisition of interactive skills. In the task, participants were told that they were in a room and they had to choose one of the two colors presented on the screen to go to the next room. After making two choices, participants would either reach an exit or a dead-end. Participants were instructed to choose the colors that would lead them to the exit as often as possible. Figure 1 shows an example of the task. In room 1, if they chose “red” they would go to room 2 with probability 0.8 and to room 3 with probability 0.2. (We will explain the objects “computers” and “books” later.) The probabilities were reversed if “blue” was chosen. After the first choice, if participants were in room 2, if they choose “yellow” there was a 0.6 probability of going to an exit and 0.4 probability of going to a dead end. Again, the probabilities were reversed if “green” was chosen. If participants were in room 3, choosing “yellow” would lead to an exit with probability 0.2 and to a dead end with probability 0.8. Choosing “green” would lead to an exit with probability 0.4 and that to a dead end with probability 0.6. Note that if “red” was chosen, “yellow” was more likely to lead to an exit than “green”; but if “blue” was chosen, “green” was more likely than “yellow”. The two choices were therefore interdependent: The more likely colors in the second choice were dependent on the first choice. We assumed that the declarative memory system would learn by a goal-directed “treesearching”. Specifically, learning would occur by encoding the first and second choices and their outcomes (exit or dead-end) in declarative memory. If they led to an exit, both choices would be credited, otherwise both would be penalized. With enough experiences, the pair of choices that would more likely lead to the exit would be chosen. The advantage of this strategy was that the credit assignments were fast and efficient, but in the expense of higher memory load. We therefore hypothesized that with the presence of a secondary task, learning by the declarative memory system would be hampered as it would be more difficult to maintain the two choices in declarative memory. To distinguish between the two learning processes based on behavioral data, the probabilities were chosen such that one of the branches would more likely lead to an exit than the other branch (in Figure 1, the “red” branch would lead to an exit with probability 0.46 while that for the “blue” branch was 0.34). It can be easily shown that no matter which color was chosen in the second choice, the probability that choosing “red” in the first choice would eventually lead to an exit would be higher than that for choosing “blue” (the marginal probabilities were 0.46 and 0.34 for choosing “red” and “blue” respectively). On the other hand, if a color was randomly chosen in the first choice, the probabilities that choosing “yellow” or “green” in the second choice would lead to an exit would be equal (the marginal probabilities were both 0.4). Given that we assumed that both choices were credited at the same time if learning occured through declarative memory encoding, we predicted that in situations where declarative memory encoding was dominant, learning the first choice would be faster than that in the second choice. Reinforcement learning was assumed to be a learning process that was distinct from declarative memory encoding. The basic prediction of reinforcement learning is that, when feedback is received after a sequence of actions, only the last cue-action association in the sequence will be reinforced (either positively or negatively) initially. As experience of different action outcomes accumulates in a specific probabilistic reward environment, the cue-action associations will gradually become secondary reinforcers such that earlier actions that lead to these cues will be reinforced. This backward propagation mechanism will gradually assign the delayed feedback to earlier cue-action associations (Fu, & Anderson, 2006; Sutton, & Barto, 1998). The major characteristic of this form of reinforcement learning is that learning of actions closer to the feedback will be faster than actions farther away from the feedback. Given that the learning process does not require active maintenance of previous actions in declarative memory, we hypothesize that it will be less hampered by a secondary task that imposes high working memory load. The design in this task allowed us to directly pit the two learning processes against each other by comparing which of the two choices would be learned faster in different conditions: When declarative memory encoding was dominant, learning of the first choice would be faster; when reinforcement learning was dominant, learning of the second choice would be faster. We predicted that in the single-task condition, declarative memory encoding would be dominant, and learning the first choice would be faster than the second choice. In the dual-task condition, we predicted that a concurrent secondary task would significantly suppress the declarative memory encoding process but not the reinforcement learning process. Thus, learning of the second choice would be faster than the first choice. In addition, we predicted that reinforcement learning was possible only when the external cues were present to distinguish between the rooms (i.e., “computers” and “books” in room 2 and 3 as shown in Figure 1). When these objects were taken away, reinforcement learning would fail, as the probability of reaching an exit after choosing yellow and green would be the same and thus no learning could occur at the second choice. Given that learning of the first choice depended on the second choice, learning of the first choice would also fail. Our second prediction on the nature of the two learning processes may seem counterintuitive. We assumed that learning by declarative encoding, as described in Figure 2, was more accessible for recall and eventually lead to top-down, rule-based behavior (e.g., Anderson, 1982). For example, in the example shown in Figure 1, learning by declarative memory encoding would more likely lead to rule-based behavior such as “if in the first room, choose Red”, or “If computers are present, choose Yellow”. This declarative rule-based behavior was found to be pervasive in probabilistic classification task, especially when the stimuli were distinguishable by a single dimension (e.g., Ashby, Alfonso-Reese, Turken, & Waldron, 1998; Ashby, & Gott, 1988). These rules were also found to be optimal to the global properties of the task (Ashby, Maddox, & Bohil, 2002). We predicted that this kind of rule-based behavior would be less affected by short-term fluctuations in its correctness in a probabilistic environment (as indicated by trial-by-trial feedback). In other words, we predicted that learning by declarative memory encoding would lead to fewer changes in their color choices after an error feedback. In contrast to declarative memory encoding, reinforcement-learning is mostly driven by gradual accumulation of experiences through trial-and-trial feedback to inform the correctness of the choices. It is therefore more likely influenced by local fluctuations of reinforcement rates (i.e., probabilities that a particular choice of colors will lead to an exit averaged over a small number (<10) of trials). Although we did not directly manipulate the distributions of local probabilities, previous studies that aimed at teasing apart the different effects of local and global probabilities have shown that people are indeed sensitive to these local probabilities (e.g., Friedman et al., 1964; Herrnstein, & Prelec, 1991). In fact, in a number of animal studies where the reinforcement learning process was believed to be dominant, sensitivities to local reinforcement rates were found to be much higher than the global reinforcement rates (e.g., Vaughan, 1981). Elsewhere, we have also shown that a model of reinforcement learning does exhibit this property of higher sensitivity to local reinforcement rates (Fu, & Anderson, 2006). We therefore predicted that the reinforcement learning process would lead to more changes in their color choices after an error feedback, compared to learning by declarative memory