Millionaire: A Hint-guided Approach for Crowdsourcing

Modern machine learning is migrating to the era of complex models, which requires a plethora of well-annotated data. While crowdsourcing is a promising tool to achieve this goal, existing crowdsourcing approaches barely acquire a sufficient amount of high-quality labels. In this paper, motivated by the “Guess-with-Hints” answer strategy from the Millionaire game show, we introduce the hint-guided approach into crowdsourcing to deal with this challenge. Our approach encourages ∗ indicates equal contributions. † indicates the corresponding author. Bo Han Centre for Artificial Intelligence (CAI), University of Technology Sydney, Australia & Center for Advanced Intelligence Project, RIKEN, Japan E-mail: [email protected] Quanming Yao 4Paradigm Inc., Beijing, China E-mail: [email protected] Yuangang Pan Centre for Artificial Intelligence (CAI), University of Technology Sydney, Australia E-mail: [email protected] Ivor W. Tsang Centre for Artificial Intelligence (CAI), University of Technology Sydney, Australia E-mail: [email protected] Xiaokui Xiao Department of Computer Science, National University of Singapore, Singapore E-mail: [email protected] Qiang Yang Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong E-mail: [email protected] Masashi Sugiyama Center for Advanced Intelligence Project, RIKEN, Japan & Graduate School of Frontier Sciences, The University of Tokyo, Japan E-mail: [email protected] ar X iv :1 80 2. 09 17 2v 2 [ cs .H C ] 6 M ar 2 01 8