Simulation, and simulation experiments

systems perspective' should help the overcoming of artificial subject barriers such as this. The work of Blunt, Duquet and Luckie was concerned with the extent of the resources needed, and the response time, of an information retrieval service, and appears to be especially useful in determining the extent to which response time is affected by the competition by queries for resources. Hertz et al. and Fried examined the problem of simulating indexed document record files. Heine was concerned with using simulation to predict the effect on Recall-Precision performance of using document age as a component of the query in addition to the more conventional semantic attributes of documents. Lastly we discuss the two theses by M. D. Cooper and Griffiths. Cooper was essentially concerned, in the simulation part of his study, with the extent to which different queries retrieved different numbers of items from a database. Pseudo-queries and pseudo-documents were defined, each as sets of document attributes. The similarity of a query with a document was expressed as the number of attributes in common between them, i.e. as coordination level (to use the Cranfield concept and terminology), and the distribution of the database over non-zero values of the latter was found for a wide variety of queries. Cooper's work appears to be notable for (a) the careful placing of the study in the context of retrieval system evaluation, and (b) the incorporation of term association (i.e. pairwise dependence between terms in some subset of the database, in this case the entire database) in the simulation. However, as stated by him the simulation is limited in its usefulness in that the notion of a partitioning of the database, and in particular a partitioning of the retrieved set, into relevant and non-relevant subsets, is not recognized in the model. The possibility of doing so was rejected by him on the ground that 'not enough information is available to characterize the process' (p. 156). This apparently minor point is dwelt on here because in the writer's view it illustrates the occasional critical dependence of simulation upon experimental results (obtained in the laboratory or from operational systems) as well as upon the system description. Griffiths, like Cooper, was concerned with creating pseudodocuments and pseudo-queries in order to simulate the process of postcoordinate searching a database. Unlike Cooper, who chose not to model users' relevance judgements or retrieved sets, Griffiths partitioned the retrieved sets arising in the simulation (identified by a matching process + threshold) by using experimental data obtained from an INSPEC test on retrieval strategies carried out in 1974, and an EEC study of databases containing veterinary literature. The simulation procedure apparently labelled retrieved documents (attached to each co-ordination level) as either relevant or non-relevant on the basis of the value of a Bernoulli variable, but a detailed description of this step and justification of it are unfortunately not given. No attempt was made to model the relevance values of non-retrieved documents (i.e. to partition non-retrieved documents into relevant nonretrieved and non-relevant non-retrieved) so that only Precision, not Recall, is modelled. Co-occurrence frequencies of terms are also not introduced into the model (unlike Cooper's model), presumably because empirical evidence was not available in support of this. Although the main objective of Griffiths was to simulate post-coordinate searching using data obtained from Some previous work in simulation applied to information retrieval 195 operational systems (where Cooper was concerned with more hypothetical data) this objective does not appear to have met with complete success, since (a) operational data for a full validation of the simulation model was not obtainable, and (b) the data that was obtainable from existing small test collections was either inadequate or inapplicable (p. 11). The main goal of preparing an information retrieval system simulation appropriate to operational retrieval systems appears therefore to be far from complete, since even if a model incorporating valid real data in all significant components could be found, there would still remain the problem of 'designing this in' to a larger model taking into account the motivations of users and supporting agencies, as discussed by Reilly, and Baker and Nance. If valid data cannot be obtained (which seems unlikely), then there is a limitation here in principle to the usefulness of the simulation approach.