Judgement Swapping and Aggregation

We present the results of an initial experiment that indicates that people are less overconfident and better calibrated when they assign confidence levels to someone else’s interval judgements (evaluator confidences) compared to assigning confidence levels to their own interval judgements (judge confidences). We studied what impact this had on a number of judgement aggregation methods, including linear aggregation and maximum confidence slating (MCS). Using evaluator confidences as inputs to the aggregation methods improved calibration, and it improved hit rate in the case of MCS.