Evaluation of MIR systems requires honesty and skepticism with respect to the selection of test collections and the interpretation of results. We describe three minimal criteria which help to ensure this honesty and skepticism through appropriate selection of test collections, elimination of bias, and objective analysis of results.

We propose scoring rule mechanisms for eliciting honest responses to a multiple choice question, MCQ. The respondent’s score consists of two terms: an auxiliary score based on his response to a related auxiliary question, and the corresponding scores of other respondents giving the same response as him to the MCQ. For this to work, the auxiliary score has to be truth-incentive for the auxiliary...

In the study of signaling, it is well known that the cost of deception is an essential element for stable honest signaling in nature. In this paper, we show how costs for deception can arise endogenously from repeated interactions between individuals. Utilizing the Sir Philip Sidney game as an illustrative case, we show that repeated interactions can sustain honesty with no observable signal co...