Estimating Reliability and Bias from Reinterviews with Application to the 1998 Dress Rehearsal Race Question

1. Introduction Reinterview surveys have been used extensively in census evaluations since 1940's. Reinterview surveys are designed with the objective of estimating measurement reliability, measurement bias, or both bias and reliability. For estimating reliability, a test-retest reinterview design is typically used. In a test-retest reinterview survey, a subset of the questions from the original survey are reasked to form a second set responses to these questions. Estimates of measurement reliability can be computed based upon the patterns of agreement or disagreement between the two responses. There are two key assumptions made for these estimates. One assumes that the error distributions for responses to the interview and reinterview are identical. This assumption implies that the reinterview survey be conducted using the same mode of interview, the same question wording, interviewers with similar training and expertise, and that it interviews the same respondents as interviewed in the original interview. A key assumption for test-retest reinterview is that measurement errors in the reinterview are independent from the original interview. This assumption usually implies that the reinterview be conducted long enough after the original interview that correlations between the errors due to respondent memory effects are minimized. A key assumption for " gold-standard " reinterview design is that the responses produced are essentially free of measurement error and, thus, deviations between the interview and reinterview responses are interpreted as errors in the original interview. The gold standard assumption requires that the reinterview use the most experienced and competent interviewers, the most preferred mode of interview, and questions which elicit highly accurate responses. Probing questions may be used to clarify responses. Further, discrepancies between the original interview reinterview responses may be reconciled (see Forsman and Schreiner, 1991, for a discussion of reinterview survey design). Often for census evaluation programs, reinterview surveys serve a number of research objectives which lead to compromises in the reinterview design. As a consequence, neither the assumptions for test-retest or gold-standard reinterview are met. For example, quite often the evaluation survey is the post-enumeration survey (PES) used for evaluating census coverage. While the census is conducted by mail, self-administered mode, the PES is usually conducted by face to face or telephone with interviewer-assistance. Moreover, since the objective of the PES is evaluation of census coverage error, the data collection procedures employed for the PES are often quite different from those used for the census. Because the PES data are collected by …