Optimal Correction for Guessing in Multiple-Choice Tests

MCQ examinations are frequently used but there is no consensus as to whether a penalty for wrong answers should be used or not. For example, in some countries examinations for medical licensing include MCQ sections with penalty while in others there is no penalty for wrong answers. We contribute to this discussion with the analysis of the effects of penalties. In a partial knowledge model there is a trade-off between bias and measurement error that is affected by the level of penalties. We build on IRT and introduce students’ optimal behavior. Our simulations indicate that the optimal penalty may be relatively high: even though it discriminates against risk averse students this effect is small compared with the measurement error that it prevents. Financial support from MEC(SEJ2006-06309/ECON) and Gobierno Vasco, DEUI(IT-313-07) is acknowledged. Mailing address: Dpto. Fundamentos del Análisis Económico II, Avda. Lehendakari Aguirre 83, 48015 Bilbao, Spain. E-mail: [email protected] [email protected] ”Religion, politics and formula scoring are areas where two informed people often hold opposing ideas with great assurance.” (Lord, 1975, p. 7) Multiple-choice tests have some advantages over constructed-response tests, such as a wider sampling of content and the prevention of grading errors. On the other hand, guessing behavior on the part of examinees affects the scores obtained in these tests. To avoid this disadvantage of multiple-choice tests, a correction for guessing formula is sometimes used. Penalizing incorrect answers reduces the incentive for guessing and subjects may leave items unanswered. Educational and psychological researchers use Item Response Theory (IRT) to evaluate tests and item scores based on the mathematical relationship between abilities and item responses. Building on IRT, we propose a theory of students’ behavior in multiple-choice tests that might explain why they may find it optimal to answer some items and leave others blank. Suppose a test has N items and each item has M possible choices. Different methods of correction for guessing have been proposed: elimination/inclusion scoring and confidence or probabilistic scoring, among others (e.g., Budescu & Bar-Hillel, 1993; Collet, 1971). Although the model developed in this paper allows for other types of scoring, we will focus on the most used scoring formula. The score, s, is a function of the number of rights, r, wrongs, w, and the penalty for wrongs, p: s = r− pw with p = 1 M−1 . Under these conditions, the expected value of a pure guess, that is getting the right answer (s = 1) with probability 1 M and failing (s = − 1 M−1) with probability M−1 M , is equal to the expected value of omitting (s = 0), that is, ( 1× 1 M )