Statistical Mimicking 1 Statistical Mimicking of Reaction Time Data: Single Process Models, Parameter Variability and Mixtures

Statistical mimicking issues involving reaction time measures are introduced and discussed in this article. Often, discussions of mimicking have concerned the question of the serial vs. parallel processing of inputs to the cognitive system. We will demonstrate that there are several alternative structures that mimic various existing models in the literature. In particular, single process models have been neglected in this area. When parameter variability is incorporated into single process models, resulting in discrete or continuous mixtures of reaction time distributions, the observed reaction time distribution alone is no longer as useful in allowing inferences to be made about the architecture of the process that produced it. Many of the issues are raised explicitly in examination of four different case studies of mimicking. Rather than casting a shadow over the use of quantitative methods in testing models of cognitive processes, these examples emphasize the importance of examining reaction time data armed with the tools of quantitative analysis, the importance of collecting data from the context of specific process models, and also the importance of expanding the data base to include other dependent measures. Statistical Mimicking 3 Statistical Mimicking of Reaction Time Distributions Since the publication of Donders' essay, “On the speed of mental processes” (1868/1969), psychologists have measured the time required by experimental subjects to perform various tasks. These reaction times (RTs) and the changes in RT under different experimental manipulations have been used as evidence for or against models of mental architecture the arrangement of the mental processes underlying the subject’s performance (Sternberg, 1969a; Townsend & Ashby, 1983; Woodworth, 1938, Chapter 14). RT data has played an important role in distinguishing between models and in testing hypotheses about processes and structures. Consequently, considerable effort has been devoted to the refinement of RT measures, from techniques to optimize the accuracy of subsequent statistical analyses of RT summary statistics (Ratcliff, 1993; Townsend, 1990b; Ulrich & Miller, 1994), to the estimation of RT distributions and RT hazard functions (Burbeck & Luce, 1982; Luce, 1986; Ratcliff & Murdock, 1976). The concentration on the RT distributions can be seen as an advance from the use of less informative summary statistics such as the mean or median. In the distributional approach, the density or distribution functions predicted by various models are fit to the usually unimodal, positively skewed RT densities or distributions produced by experimental subjects (e.g., Green & Luce, 1971; Heathcote, Popiel, & Mewhort, 1991; Hockley, 1984; Hohle, 1965; McGill & Gibbon, 1965; Ratcliff, 1978, 1979, 1988; Ratcliff & Murdock, 1976). The success or failure of the fitting process indicates the appropriateness of a particular model for the task at hand. In this paper, we wish to address the issue of statistical mimicking of RT data; that is, the ability of very different kinds of models to produce similar patterns of mean RTs and RT distributions. Very often, highly dissimilar mental architectures can produce RTs that are indistinguishable from each other, at least in the sense that appropriate statistical analyses applied to the data cannot determine any differences between the RT distributions. The existence of such statistical mimics to various models of performance raise concerns about the way that various tests proposed for RT analyses, especially when these tests are applied without the constraints of processing models. We will discuss several such tests in the first half of the paper, and examine how they are able to Statistical Mimicking 4 distinguish between data generated by different kinds of models. In the second half of the paper, we will consider two models of RT performance which rely on multiple stages of processing, and demonstrate that a very different kind of model that does not rely on multiple processes can also fit the RT data. In so doing, we emphasize that analyses of RT alone are not sufficient to distinguish between these types of models. Additional measures, such as accuracy or confidence judgments, and the observations of the behavior of RT distributions over a range of experimental conditions, are needed to determine the adequacy of these models of performance. We must emphasize that the issue of model identifiability is not limited the RT paradigms that we discuss here, nor to the area of cognitive psychology in general. This problem will arise across the different disciplines and is worst for those areas that have not benefitted, as cognitive psychology has, from concentrated attempts to quantify psychological findings. Demonstrating that this issue is still a concern for cognitive psychology underscores the problem for those other areas. As cognitive psychologists, we work in the areas with which we are most familiar, but this should not be taken as a signal that other areas of experimental psychology are exempt from the issues we raise. We will begin by outlining the mimicking problem, and a solution that has been proposed to circumvent it. We will then discuss the issue of parameter variability, which undermines the utility of that solution. Later in the paper, these topics will be addressed concretely with four case studies, two concerning model free tests of processing and two concerning specific multiprocess models. In each of these case studies, we will present an alternative single process model that either passes the test for a multiprocess model or accounts for RT data as well as the multiprocess model. Under no circumstances should these findings be interpreted as a demonstration of weakness in the tests. Rather, they demonstrate that there is a right way and a wrong way to apply them. Mimicking RT data has been collected for a wide variety of experimental paradigms. These data have been used to address questions concerning the serial or parallel operation of the processes involved in the task of interest, the nature of information transmission from Statistical Mimicking 5 one process to the next, and the hierarchical organization of the processes. The issue of serial vs. parallel arrangement of the processes in memory retrieval initially received a great deal of attention (e.g., Sternberg, 1966; Townsend, 1972, 1974; Townsend & Ashby, 1983). This was due in part to the nature of the paradigms and stimulus materials used in memory “search” experiments. For instance, an experiment designed to test some hypothesis about memory function typically has subjects learn a list of words or other items and then present them with a test item to which they should answer “old” or “new.” A natural first question concerning the way the memory process operates is whether the study list items in memory are compared with the test item serially (one at a time), or in parallel (simultaneously)? Unfortunately, RTs do not readily distinguish between these two types of architectures. When a single independent variable is manipulated, such as list length, both serial and parallel models can produce identical patterns of RT. For example, for every model in which subprocesses operate in parallel and the time required to complete each subprocess has no influence on (is independent from) the amount of time required by any other subprocess, there exists a mathematically equivalent model, not discriminable from the parallel model, in which all subprocesses operate in series. The RTs produced by the independent parallel model and its equivalent serial representation will be identical in every way. Townsend's (1972) careful enunciation of this theoretical pitfall discouraged further research that relied on the premise that mean RT data alone could discriminate between serial and parallel processes in these kinds of tasks. The existence of serial mimics to parallel processes is probably the most wellknown example of an identifiability problem in cognitive modeling. The serial/parallel question itself is very specific, easily operationalized, seemingly tractable, and so at first blush it appears to be just the type of question that cognitive psychology should devote itself to answering. Townsend (1990a) argues that the serial/parallel issue is indeed exactly the kind of problem that should be resolved, but unfortunately it has not been. Despite a growing body of theoretical work outlining how RT can be used to distinguish between serial and parallel processing (e.g., Roberts & Sternberg, 1994; Schweickert, 1980; Schweickert & Townsend, 1989; Sternberg, 1969a; Townsend & Ashby, 1983; Townsend & Schweickert, 1989), empirical resolution of the serial/parallel issue seems to Statistical Mimicking 6 have fallen by the wayside. Some researchers might say that the reason for this is the question itself: since all physiological evidence suggests that processing is parallel at some level anyway, it might seem pointless to invest the effort required to debunk the serial “straw man.” Also, the paradigms that addressed the question were somewhat limited, usually involving memory or visual search (although richer paradigms have since been proposed, cf., Schweickert & Townsend, 1989). The RT methodology is now applied to other, perhaps more interesting questions for which the problem of model identifiability does not (yet) exist, such as attentional control (e.g., Treisman, Vieira, & Hayes, 1992; Wolfe, Cave, & Franzel, 1989), the nature of information flow through the system (e.g., McClelland,1979; Miller, 1988, 1993), the acquisition of skill (e.g., Carlson & Schneider, 1989; Logan, 1988, 1992), and so on. However, the way that RTs are employed to test hypotheses in other areas is often little changed from the way that they were applied to the serial/parallel question. Other performance variables are often ignored, and rigorously defined models of the processes of interest are often lacking. Therefore these areas may also be prone to the problem of mimicking between various types of models. The serial/parallel processing issue is one of exact mathematical equivalence of two models: the RTs that they predict can be identical although the structures of the models are very different. Even if exactly equivalent serial representations of independent parallel models did not exist, however, the RTs produced by the serial and parallel models might still be ambiguous. As several researchers have noted (e.g., Luce, 1986; Ratcliff, 1988), the unimodal and positively skewed RT density can easily be fit by a number of distributions. For example, the gamma, inverse normal and Ex-Gaussian distributions have all been shown to fit RT data to a greater or lesser degree (Ratcliff & Murdock, 1976). Because the shape of the distributions are highly similar, this “statistical mimicking” of RT data is still a concern even when mathematically equivalent relationships, such as those that arise in the serial/parallel case, do not exist. We now discuss issues of statistical power and the number of observations needed to discriminate between models that predict similar RT distributions. Then we will present how the hazard function can be used to provide finer discrimination between models, and how parameter variability (leading to mixtures of distributions) reduces statistical power and the diagnosticity of the hazard Statistical Mimicking 7