Pairwise choice as a simple and robust method for inferring ranking data

One of the largest challenges for a recommender system is building a ranking of “quality” or “relevance” in situations where these features cannot be observed directly. These models are often trained on various types of survey data, including Likert-scale quality ratings or pairwise comparison surveys, but there has been little work detailing the efficiency of these techniques for eliciting quality ranking and a parsity of work on how to analyze and interpret pairwise choice data. We present techniques for using pairwise choice data for quality ranking and we find, under simulation, that Likert scale elicitation is more efficient under the best possible conditions but in the presence of differential item functionality (i.e., the fact that different scale points may mean different things to different people) or low quality inputs (e.g., lack of attention or understanding by survey participants or noisily measured input features) pairwise comparison becomes a more efficient survey method. We confirm this finding by using different survey techniques to infer the relevance of individuals’ Facebook News Feed stories. Pairwise choice elicitation can be finished quickly by survey participants, is easily to implement and scale, produces models with interpretable results and is robust to noise and interpretational issues. Thus, we argue, pairwise choice surveys have wide potential for application.