Shallow pooling for sparse labels

Recent years have seen enormous gains in core information retrieval tasks, including document and passage ranking. Datasets leaderboards, particular the MS MARCO datasets, illustrate dramatic improvements achieved by modern neural rankers. When compared with traditional test collections, such as those developed TREC, datasets employ substantially more queries—thousands vs. dozens – fewer known relevant items per query—often just one. For example, 94% of nearly seven thousand queries ranking development set only a single passage, no query has than four. Given sparsity these relevance labels, leaderboards track mean reciprocal rank (MRR). In essence, item is treated “right answer” or “best answer”, rankers scored on their ability to place this high possible. working sparse we observed that top returned ranker often appear superior judged items. Others reported same observation. To observation, employed crowdsourced workers make preference judgments between stack for set. The results support our If imagine hypothetical perfect under MRR, score 1 all queries, indicate searcher would prefer result from frequently ranker, making “better perfect”. understand implications leaderboard, pooled available runs near leaderboard over 500 queries. We pools re-evaluated runs. Our concerns current may longer be able recognize genuine future, if are measured against answer, answer should best most preferred maintained ongoing judgments. Since required, maintenance might performed shallow pooling. previously unjudged surfaced ranking, it can directly previous answer.