Modular Processes in Mind and Brain Saul Sternberg* Running Head: Modular Processes in Mind and Brain
چکیده
One approach to understanding a complex process starts with an attempt to divide it into modules: subprocesses that are independent in some sense, and have different functions. In this paper I discuss an approach to the modular decomposition of neural and mental processes. A dozen examples of process decomposition are presented. Tw o are of well-established and purely behavioral realizations of the approach (signal detection theory applied to discrimination data; the method of additive factors applied to reaction-time data), and lead to the identification of mental modules. Other examples, leading to the identification of modular neural processes, use brain measures, including the latency and amplitude of electrophysiological events, and fMRI measures. Some measures are pure (reflecting just one process), while others are composite. Two of the examples reveal mental and neural modules that correspond. Tw o promising but incomplete applications are attempts to relate brain regions to behaviorally defined processing modules, using a brain manipulation (transcranial magnetic stimulation, TMS), but suffer from flaws in design or analysis. I show why the process-decomposition approach discussed here, in which the criterion for modularity is separate modifiability, is superior for modular decomposition to the more frequently used task comparison procedure (often used in cognitive neuropsychology) and to its associated subtraction method. To demonstrate the limitations of task comparison, I discuss an example involving TMS. * E-mail: saul@psych.upenn.edu Tel: 215-898-7162 Fax: 215-898-7210 Address: Department of Psychology University of Pennsylvania 3401 Walnut Street, C-Wing Philadelphia, PA 19104-6228, USA Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 2 Acknowledgements For providing unpublished details of their data I thank Brent Alsop, Stanislas Dehaene, Russell Epstein, Silke Goebel, John Kounios, Lotfi Merabet, Allen Osman, Alvaro PascualLeone, Philippe Pinel, Eric Schumacher, and Fren Smulders. For helpful discussions, I thank Geoffrey Aguirre, David Brainard, Russell Epstein, Martha Farah, Joshua Gold, Nancy Kanwisher, John Kounios, David Meyer, Jacob Nachmias, Allen Osman, and Seth Roberts. For helpful comments on the manuscript I thank Stanislas Dehaene, Silke Goebel, Allen Osman, Eric Schumacher, Fren Smulders, Sharon Thompson-Schill, Vincent Walsh, Max Coltheart, and two anonymous reviewers. For computer support I thank Vincent Hurtubise, Christopher Leary, and Roderick Smith. Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 3 Modular Processes in Mind and Brain Saul Sternberg 1. Modules and Modularity The first step in one approach to understanding a complex process is to attempt to divide it into modules: parts that are independent in some sense, and have different functions.1 Early in the last century, scientific psychology, dominated by behaviorism, emphasized the directly observable relations between stimuli and responses, and devoted little effort to describing the perception, memory, and thought processes that intervene. During the second half of the century there was a change in the kinds of questions that psychologists asked and in the acceptable answers. This change was perhaps influenced by the growth of computer science, which persuaded psychologists that programming concepts might be acceptable as precise descriptions of information processing by people as well as by machines. And the software-hardware distinction added legitimacy to theories couched in terms of abstract information-processing operations in the mind rather than only neurophysiological processes in the brain. In the "human information processing" approach, complex activities of perception, decision, and thought, whether conscious or unconscious, came to be conceptualized in terms of functionally distinct and relatively independent ("modular") subprocesses responsible for separate operations such as input, transformation, storage, retrieval, and comparison of internal representations — modules whose arrangement was expressed in systematic flow charts.2 A useful distinction that is encouraged by comparing the brain to a digital computer is that between processors and the processes they may implement. The existence of functionally specialized processors (either localized or distributed) is a sufficient condition but not a necessary one for functionally distinct processes, the concern of the present paper. The rise in the 1980s of parallel distributed processing might seem to conflict with the idea of modular organization of processes, but it need not: such models "do not deny that there is a macrostructure," and are intended to "describe the internal structure of the larger (processing) units" (Rumelhart, McClelland, & the PDP Research Group, 1986, p. 12). Furthermore, even starting with a relatively unstructured neural network, there is reason to believe that over time and with experience it will develop functionally specialized processing modules, and hence, functionally specialized processes (Jacobs & Jordan, 1992; Jacobs, 1999). To ward the end of the century, it became possible (using fMRI, for example) to measure the human brain in action with previously unattainable spatial resolution. Because functions of the brain are often implemented by specialized neural processors that are anatomically localized, these new measurement methods encouraged scientists to attempt the modular decomposition of complex neural processes, just as they had been doing for complex mental processes. 1. A module may itself be composed of modules. 2. Heuristic arguments for the modular organization of complex biological computations have been advanced by Simon (1962, 2005) and, in his ‘principle of modular design’, by Marr (1976). Marr (p. 485) argued that "Any large computation should be split up and implemented as a collection of small sub-parts that are as nearly independent of one another as the overall task allows. If a process is not designed in this way, a small change in one place will have consequences in many other places. This means that the process as a whole becomes extremely difficult to debug or to improve, whether by a human designer or in the course of natural evolution, because a small change to improve one part has to be accompanied by many simultaneous compensating changes elsewhere." Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 4 In the present paper I describe an approach to the modular decomposition of two kinds of complex process, mental and neural, along with several examples of its application from the psychology and cognitive neuroscience literature.3 A mental-process module is a part of a process, functionally distinct from other parts, and investigated with behavioral measures. Such modules will be denoted A, B, etc. A neural-process module is a part of a neural process, functionally distinct from other parts, and investigated with brain measures. Such modules will be denoted α , β , etc. The important distinction between processes and processors is sometimes overlooked. Processes occur over time; their arrangement is described by a flow-chart. In contrast, processors (e.g., as described by Erickson, 2001; Goodale, 1996; Op de Beeck, Haushofer, & Kanwisher, 2008) are parts of a physical or biological device (such as the brain); their arrangement can often be described by a circuit diagram.4 2. The Process-Decomposition Approach 2.1 Separate modifiability, selective influence, process-specific factors, and functional distinctness Much thinking by psychologists and brain scientists about the decomposition of complex processes appeals either implicitly or explicitly to separate modifiability as a criterion for modularity: Two (sub)processes A and B of a complex process (mental or neural) are modules if and only if each can be changed independently of the other.5 One purpose of the present paper is to explicate by example the notion of separate modifiability and the conditions under which one can assert it. To demonstrate separate modifiability of A and B, we must find an instance of selective influence. That is, we must find experimental manipulations (factors) F and G that influence A and B selectively, i.e., such that A is influenced by F but is inv ariant with respect to G, whereas B is influenced by G but is inv ariant with respect to F. Often one starts with hypotheses about what the component processes are, and about corresponding process-specific factors that are likely to influence them selectively. Alternatively, the selectivity of effects may be discovered in experiments with other goals. Separate modifiability of A and B is also evidence for their functional distinctness; information about what a process does is provided by the sets of factors that do and don’t influence it; if two processes had the same function they would be influenced by the same factors.6 2.2 Processes and their measures, pure and composite, and combination rules How do we demonstrate that a process is influenced by a factor, or inv ariant with respect to 3. In Sternberg, 2001, referred to as "SM" or "SM:[Section]", I discuss and defend the approach in more detail, describe its antecedents, illustrate it with several applications to mental and neural processes, and further discuss its inferential logic. 4. It is perhaps a confusion between time (process) and space (processor) that has led some commentators (Broadbent, 1984, and one of the reviewers of this paper) to believe that a process organized in stages cannot include feedback because it is implemented by a "pipeline": an ordered set of processors through which information passes in a fixed direction. 5. This criterion for modularity seems to be far weaker than the set of module properties suggested by Fodor (1983), according to whom modules are typically innate, informationally encapsulated, domain specific, "hard-wired", autonomous, and fast. However, domain specificity appears to imply separate modifiability. 6. Such double dissociation of subprocesses should be distinguished from the more familiar double dissociation of tasks (Schmidt & Vorberg, 2006), discussed in Section 9. Sternberg (2003) discusses the distinction; see also SM:1.1). Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 5 it? We know only about one or more hypothesized measures MA of process A, not about the process as such. Depending on the available measures, there are two ways to assess separate modifiability of A and B. Table 17 Inferential logic for pure measures. Joint Hypothesis H1: Processes A and B are modules (separately modifiable). H2: MA, MB are pure measures of A, B. Prediction We may be able to find factors F and G that influence MA(A) and MB(B) selectively: p1: MA ← F , p2: MB ←/ F , p3: MB ← G, p4: MA ←/ G. Alternative Results We find factors F , G that influence MA and MB selectively. We fail to find such factors. Inferences Support for joint hypothesis Refutes one/both of H1, H2, or H1 + H 2. we didn’t look enough for F , G. Pure measures. Suppose we have pure measures MA and MB of the hypothesized modules: A pure measure of a process is one that reflects changes in that process only. Examples include the durations of two different neural processes (see Section 3.1), and the discriminability and criterion parameters of signal-detection theory (which reflect sensory and decision processes; see Section 5). To show that F and G influence A and B selectively, we must demonstrate their selective influence on MA and MB. That is, we must show that MA is influenced by F and invariant with respect to G, and vice versa for MB . If F j has two lev els, j = 1, 2, the effect of F on MA is a difference: (1) effect(F) = M A(F2) − M A(F1) . For factors with multiple levels the effect can be regarded as a vector of differences associated with successive ordered levels. The logic for inferring separate modules when we hypothesize that we have pure measures is shown in Table 1.8 The {pk} are four properties of the data; MA ← F should be read as "MA is influenced by F"; MB ←/ F should be read as MB is not influenced by F . All four of the requirements are critical; the instances of invariance are meaningful only if we also know that both factors are potent and both measures are sensitive. Unfortunately, it is seldom appreciated that persuasive evidence for invariance cannot depend solely on failure of a significance test of an effect: such a failure could merely reflect variability and low statistical power.9 7. Table 2 of Sternberg, 2001; reprinted by permission. 8. When the hypotheses about A and B are sufficiently detailed to specify particular process-specific factors that should influence them selectively, this leads to an alternative formulation of the inferential logic, in which the specification of F and G is included in the joint hypothesis, with the remainder of the reasoning adjusted accordingly. For a discussion of such alternatives, see SM:A.2.3. Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 6 Composite measure. Suppose that instead of pure measures, we have a composite measure MAB of the hypothesized modules — a measure to which they both contribute. Examples of composite measures are the event-related potential (ERP) at a particular point on the scalp (which may reflect several ERP sources in the brain), and reaction time, RT (which may depend on the durations of more than one process of interest). To support a hypothesis of selective influence in this case we must also know or hav e evidence for a combination rule — a specification of how the contributions of the modules to the measure combine. With pure measures, factorial experiments (rather than separate experiments for different factors) are desirable, because they provide efficient tests of generality and promote new discoveries at little cost. With a composite measure factorial experiments are essential, to assess how the effects of the factors combine; unfortunately they are rare, despite their efficiency (Section 12.5). Table 210 Inferential logic for a composite measure with summation as the combination rule. Joint Hypothesis H1: Processes A and B are modules (separately modifiable). H3: Contributions uA, vB of A, B to MAB(A, B) combine by summation. Prediction We may be able to find factors F and G that influence A and B selectively: p′ 1: uA ← F , p′2: vB ←/ F , p′3: vB ← G, p′4: uA ←/ G, and jointly influence no other process. If so, their effects on M AB will be additive. Alternative Results We find factors F and G with additive effects on M AB . We fail to find such factors. Inferences Support for joint hypothesis Refutes one/both of H1, H3, H1 + H 3. or we didn’t look enough for F , G. The logic for inferring separate modules using a composite measure when we either know or hypothesize that the combination rule is summation is shown in Table 2.11 To understand Table 9. A common error of interpretation is to assert the nonexistence of an effect or interaction merely because it fails to reach statistical significance. In evaluating a claim that an effect is null, it is crucial to have at least an index of precision (such as a confidence interval) for the size of the effect. An alternative is to apply an equivalence test that reverses the asymmetry of the standard significance test (Berger & Hsu, 1996; Rogers, Howard, & Vessey, 1993). In either case we need to specify a critical effect size (depending on what we know and the particular circumstances) such that it is reasonable to treat the observed effect as null if, with high probability, it is less than that critical size. The critical size might be determined by the sizes of effects generated by plausible models. An example of suitable caution about a null effect can be found in Ghorashi, Enns, Klein, and Di Lollo (2010). 10. Table 3 of Sternberg (2001); reprinted by permission. Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 7 2 it is important to keep in mind what the effect of a factor is, and what it means for effects of different factors to be additive. To simplify the discussion, let us assume that there are two factors, each with just two lev els. Let u and v be the contributions of processes A and B to MAB. If summation is the combination rule, we have MAB = u + v. If A and B are selectively influenced by factors F and G, we hav e (2) MAB(F j, Gk) = u(F j) + v(Gk) , where u(F j) is a function that describes the relation between the the level of F and the contribution of A to MAB. Now in general, we work with averages rather than individual values of MAB, and we regard u, v, and MAB as random rather than deterministic variables. Using M , u, and v to indicate the means of these random variables, it is convenient that with no further assumptions12 , Eq. 2 implies: (3) MAB(F j, Gk) = u(F j) + v(Gk) . In what follows, I treat the levels of factors as ordered, which permits describing changes in level as increases or decreases. From Eq. 2 it is easy to show that F and G are additive factors: the combined effect on MAB of increasing the levels of both F and G is the sum of the effect of increasing only F and the effect of increasing only G: (4) effect(F , G) ≡ MAB(F2, G2) − MAB(F1, G1) = [u(F2) + v(G2)] − [u(F1) + v(G1)] = [u(F2) − u(F1)] + [v(G2) − v(G1)] ≡ effect(F) + effect(G) . Eq. 2 also implies that the effect of each factor will be invariant over lev els of the other. Thus, (5) effect(F | G = Gk) ≡ MAB(F2, Gk) − MAB(F1, Gk) = [u(F2) + v(Gk)] − [u(F1) + v(Gk)] = [u(F2) − u(F1)], regardless of Gk . A giv en measure may be pure or composite, depending on the factors being varied and the hypothesized modules of interest. This attribute of a measure is part of the joint hypothesis that is tested as part of the process-decomposition approach.13 2.3 Overview of examples and issues Much of what follows consists of descriptions of ten examples of the process decomposition approach, the first eight successful. There are also brief summaries of two additional examples, in Section 10, which are included to convey the breadth of the approach. Examples will be referred to by the section numbers in which they are first discussed. Thus, the example discussed in Section 4.3 will be called "Ex. 4.3". With two exceptions, the successful primary examples involve factorial experiments with two factors. The exceptions are Ex. 3.2, in which the effects of the two 11. Whereas properties { pk} (Table 1) apply to observable quantities, the analogous properties { p′k} (Table 2) apply to contributions to a composite measure that are not directly observable. 12. Matters are not so simple for other combination rules, such as multiplication; see Section 8.2. 13. The reasoning is sometimes erroneously expressed as "If we assume H3 then additivity confers support on H1." This ignores the support that additivity also confers on H3. Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 8 factors are studied in separate experiments, and Ex. 4.3, in which the effects of three factors are considered. In all cases the factors have been selected because it is hoped that they will be "process specific": that they will selectively influence only one of the two or more processes that are hypothesized to underlie performance of the task. The final two examples (10.1 and 10.2) are intriguing and tantalizing cases where the approach could have been used but was not, because of incompleteness of either the design or the analysis. In Section 3 I discuss two applications based on electrophysiological measurements at the scalp from which pure measures are derived. In both cases, two neural modules are identified: In Ex. 3.1, in which the modules are associated with preparation of two different aspects of the response, they are found to operate successively, as stages. In Ex. 3.2, in which the modules are associated with encoding two different aspects of the stimulus, they are found to operate in parallel. The RT data from Ex. 3.1 are discussed in Section 4.1, where the RT is treated as a composite measure and shown to lead to the identification of two mental modules that correspond to the neural modules inferred from the electrophysiological data discussed earlier. (By "correspond to" I mean that their durations are influenced selectively by the same factors and to the same extent.) The inference from the RT data exemplifies the method of additive factors (AFM), discussed more generally in Section 4.2, and applied in Section 4.3 to the problem of locating the effect of a manipulation of interest (sleep deprivation) within a pair of already established mental modules, one for encoding the stimulus, the other for selecting the response. As discussed in Section 5, signal detection theory has provided a widely applied method for measuring sensory processes in tasks that also involve decision processes, but has in general failed to isolate those decision processes from sensory factors. Ex. 5 illustrates a variant of the method that succeeds, and thereby demonstrates the modularity of sensory and decision processes, in an experiment with pigeons in which the decision factor is not the traditional payoff matrix. To the extent that there is localization of function in the brain, so that each of several modular processes are implemented in disjoint regions, and to the extent that the level of activation in a region varies with changes in the process it implements, the level of such activation can function as a pure measure of the process. In Sections 6.3 and 6.4 I discuss two examples in which fMRI signals in different brain regions were measured for this purpose; in both cases, RTs were measured as well. In Ex. 6.3 (number comparison), in which RT measurements in a similar experiment had already indicated separate stages for encoding the test number and comparing it to the target, both the new RT data and the fMRI data support this analysis, suggesting mental and neural modules that correspond. However, the fact that the direction of the effect of the encoding factor on the fMRI response differs in different brain regions raises interesting questions of interpretation. In Ex. 6.4 (manual choice-reaction with four stimulus-response pairs) the fMRI data support the hypothesis of modular neural processes for stimulus encoding and response selection, but unlike earlier observations of effects on RT from several similar paradigms, the effects of the two factors on RT interact rather than being additive, raising questions of interpretation. One important finding in this example is the additivity of effects of the encoding and response-selection factors on the fMRI measure in the two Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 9 brain regions where both factors were found to have effects. The examples in Sections 6.5 and 8.1 use the fMRI adaptation method; in both cases the measure, which reflects the dissimilarity between the current percept and previous ones, is composite, raising the issue of the combination rule. Ex. 6.5, concerned with the perception and memory of scenes, shows that different processes are responsible for immediate and delayed adaptation effects in the parahippocampal place area. Perhaps because factorial experiments are relatively rare in studies of brain activation, we have had little experience in considering the implications of the additivity of effects on composite fMRI measures. It is tempting to infer that modular brain mechanisms are responsible. In Section 7 I consider the conditions under which inferences from additivity to modularity are justified, for both RT and fMRI. In Section 8 I provide brief summaries of three additional examples of the process decomposition approach, using different composite measures. Ex. 8.1 uses the fMRI adaptation method also used in Ex. 6.5 to test the idea that perceptually separable dimensions are encoded by different neural processes, whereas perceptually integral dimensions are not. One reason for including Ex. 8.2, a classic study that provides evidence for modular spatial-frequency analyzers in visual pattern detection, is that unlike the other examples that use composite measures, the combination rule here is multiplication rather than summation. Ex.8.3 illustrates the power of using the amplitude of the event-related potential as a composite measure. In Section 9 I contrast the process-decomposition approach with the taskcomparison method. Here, inferences are drawn from the effects on different tasks of factors such as the presence versus absence of sleep deprivation or the presence or absence of a lesion in a particular brain region, together with theories about the processes used to accomplish those tasks. I describe two examples of task comparison. One is based on the effects of repetitive transcranial magnetic stimulation (TMS) on two kinds of tactile perception. The other is Donders’ subtraction method, originally developed for RT experiments, but, in recent years, applied to brain-activation data. Unlike task comparison, which is often used in a way that requires various assumptions (including modularity) to be made without test (Shallice, 1988, Ch. 11; SM:A.1), the processdecomposition approach incorporates such tests. In Section 9.3 I describe the conflict between the conclusion from Ex. 4.3 that the effect of sleep deprivation is selective and a claim, based on task comparison, that the effect is global. The final two examples involve the effects of TMS on RT measurements. The goal is to relate brain regions to mental modules. In these experiments, TMS in certain brain regions is found to increase RT without otherwise disrupting performance. The presence and absence of TMS in such a region can be regarded as two lev els of a factor. If this factor influences RT , the brain region to which TMS is applied probably plays some role in performance of the task. However, the potential of this way of associating brain regions with modular subprocesses is realized only when the TMS factor is used together with other, process-specific factors, to determine which of their effects, if any, are modulated by TMS. For visual search (Ex. 10.1), the absence of variation of a withintask search-specific factor precluded doing so. For number comparison (Ex. 10.2) a comparison-specific factor was varied, but no focused test of the interaction of this factor Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 10 with TMS was conducted. The traditional task of cognitive neuropsychology is to learn about the cognitive functions of the normal brain by studying what happens to them when it is damaged, typically by comparing the effects of damage in different tasks. Can brain damage also be used in the process decomposition approach? This question is considered in Section 11; several reasons lead to a negative answer. Instead, the two approaches can inform each other. Several additional issues are considered briefly in Section 12. 3. Decomposing Neural Processes with the Lateralized Readiness Potential 3.1 Serial modules (stages) for preparing two response features Consider a trial in a choice-reaction experiment where two alternative responses are made by the two hands. Recall that the part of the motor cortex that controls a hand is contralateral to that hand. When enough information has been extracted from the stimulus to permit selection of the hand, but before any sign of muscle activity, the part of the motor cortex that controls that hand becomes more active than the part that controls the non-selected hand. This asymmetric activity can be detected as an increase in the av erages over trials of the difference between electrical potentials (ERPs) at the two corresponding scalp locations.14 Let Amc(t) (an index of motor-cortex asymmetry) express this difference as a function of time after stimulus onset (if "stimulus locked"), or as a function of time before the overt response (if "response locked"). Amc(t) is normally zero, but is defined to as to become positive when the (correct) response hand is selected; the increase of such asymmetry is called the lateralized-readiness potential (LRP). The onset time of the LRP is thus an estimate of the time at which the side of the response (left or right) has been selected. Having an estimate of this time makes it possible to ask whether the neural process responsible for selecting the side of the response is separate from the neural process responsible for preparing other aspects of the response. This possibility was exploited in an experiment by Smulders, Kok, Kenemans, and Bashore (1995). It was a two-choice RT experiment with single-digit stimuli mapped on left-hand and right-hand responses. Two factors were varied orthogonally across blocks of trials: stimulus quality (SQ j : digit intact versus degraded) and response complexity (RCk : one keystroke — simple — versus a sequence of three keystrokes — complex — made by fingers of the responding hand). In addition to the composite measure RT jk , Smulders et al. measured the onset time of the LRP, based on both stimulus-locked (LRPs) and response-locked (LRPr ) averaging of the Amc(t) functions. Let the LRP onset times be Tsjk (measured with respect to the stimulus) and T * rjk (a negative quantity, measured with respect to the response), and let Trjk = RT jk + T rjk . Tsjk and Trjk are then 14. As in some other brain measurements (e.g., PET, fMRI), the poor S/N ratio often means that av eraging over trials is required for the measures to be interpretable. Here, the "noise" is due partly to neural events unrelated to the task being performed, whose contributions are eliminated by combining subtraction of the pre-stimulus baseline level with an averaging process that reveals only those events that are consistently time-locked to the stimulus or the response. Sternberg Modular Processes in Mind and Brain draft 10.05.22 Page 11 different estimates of the same time point between stimulus and response, measured from the stimulus; let T⋅jk be their mean. Av eraging over the four conditions, RT⋅⋅ = 41 6 ms and T⋅⋅⋅ = 26 4 ms. If α is the neural process from stimulus to LRP onset, and β is the neural process from LRP onset to response, these values permit us to estimate their durations (pure measures of α and β ) averaged over the four conditions: α⋅⋅ = T⋅⋅⋅ = 26 4 ms and Dβ⋅⋅ = RT⋅⋅ − T⋅⋅⋅ = 15 2 ms. Stimulus Quality (SQ) Intact Degraded M ea n D ur at io n of S ta ge α (m s) 130 150 170 240 260 280