Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption is that all items in a given test set are equal with regards to difficulty and discriminating power. We propose Item Response Theory (IRT) from psychometrics as an alternative means for gold-standard test-set generati...

Embedding item response theory (IRT) models within a multilevel modeling framework has been shown by many authors to allow better estimation of the relationships between predictor variables and IRT latent traits (Adams, Wilson, & Wu, 1997). A multilevel IRT model recently proposed by Kamata (1998, 2001) yields the additional benefit of being able to accommodate data that are collected in hierar...

Two versions of a normal and lognormal model for the response times of examinees on test items are investigated. The models have a parameter structure analogous to the 2PL response models in IRT, with a parameter for the speed of each person as well as parameters for the time intensity and discriminating power of each item. It is shown how the parameters can be estimated by a Markov chain Monte...

This article analyzes latent variable models from a cognitive psychology perspective. We start by discussing work by Tuerlinckx and De Boeck (2005), who proved that a diffusion model for 2-choice response processes entails a 2-parameter logistic item response theory (IRT) model for individual differences in the response data. Following this line of reasoning, we discuss the appropriateness of I...

The purpose of this study is to develop Education Value Perception Scale (EVPS) based on Bronfenbrenner's Ecological Theory and investigate its psychometric properties according Classical Test (CTT) Item Response (IRT). data were collected from 2872 secondary school students by stratified purposeful sampling method. Measurement invariance EVPS was tested multigroup confirmatory factor analysis ...

Linear factor analysis (FA) models can be reliably tested using test statistics based on residual covariances. We show that the same statistics can be used to reliably test the fit of item response theory (IRT) models for ordinal data (under some conditions). Hence, the fit of an FA model and of an IRT model to the same data set can now be compared. When applied to a binary data set, our experi...

Modeling the interaction between persons and items at the item level for binary response data, item response theory (IRT) models have been found useful in a wide variety of applications in various fields. This paper provides the requisite information and description of software that implements the Gibbs sampling procedures for the one-, twoand three-parameter normal ogive models. The software d...

OBJECTIVE Public understanding of HPV is important to ensure informed participation in cervical cancer prevention programmes. While many studies have measured HPV knowledge, none has developed a validated measure for use across countries. We aimed to develop and validate such a measure. METHOD Items tapping knowledge of HPV, HPV testing and HPV vaccination were developed from previous literat...

We apply collaborative filtering (CF) to dichotomously scored student response data (right, wrong, or no interaction), finding optimal parameters for each student and item based on cross-validated prediction accuracy. The approach is naturally suited to comparing different models, both unidimensional and multidimensional in ability, including a widely used subset of Item Response Theory (IRT) m...

The Working Alliance Inventory (WAI, Horvath & Greenberg, 1989) has made possible tremendous contributions to psychotherapy research. Many studies support its reliability and validity, but some research suggests the 7-point response format and 3-factor structure of the client version WAI are problematic (Hatcher & Gillaspy, 2006). We used Rasch model item response theory (IRT) to analyze archiv...