Amazon's Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of resea...

Amazon’s Mechanical Turk service makes linguistic experimentation quick, easy, and inexpensive. However, researchers have not been certain about its reliability. In a series of experiments, this paper compares data collected via Mechanical Turk to those obtained using more traditional methods One set of experiments measured the predictability of words in sentences using the Cloze sentence compl...

Many crowdsourcing studies have been conducted that utilize Amazon Mechanical Turk, a crowdsourcing marketplace platform. The Amazon Mechanical Turk team proposes that comprehensive studies in the areas of HIT design, workflow and reviewing methodologies, and compensation strategies will benefit the crowdsourcing field by establishing a standard library of repeatable patterns and protocols. Author

Katarina Ivon, Od Kačića do Svačića: Imagologija i ideologija zadarskih koledara. Zagreb: Naklada Jesenski Turk, 2018.

Amazon's Mechanical Turk service has been successfully applied to many natural language processing tasks. However, the task of named entity recognition presents unique challenges. In a large annotation task involving over 20,000 emails, we demonstrate that a compet­ itive bonus system and inter­annotator agree­ ment can be used to improve the quality of named entity annotations from Mechanical ...

Crowdsourcing platforms such as Amazon Mechanical Turk have become popular for a wide variety of human intelligence tasks; however, quality control continues to be a significant challenge. Recently, we propose TURKONTROL, a theoretical model based on POMDPs to optimize iterative, crowdsourced workflows. However, they neither describe how to learn the model parameters, nor show its effectiveness...

While much work has studied crowdsourced transcription via Amazon’s Mechanical Turk, we are not familiar with any prior cross-platform analysis of crowdsourcing service providers for transcription. We present a qualitative and quantitative analysis of eight such providers: 1888-Type-It-Up, 3Play Media, Transcription Hub, CastingWords, Rev, TranscribeMe, Quicktate, and SpeakerText. We also provi...

Research and commerce activity has been expanding the potential of micro-task markets. Initially used for simple, disconnected tasks, they have now been able to achieve impressive results in creative domains such as writing and design. The goals for this research are to further explore the possibilities of micro-task markets for performing creative work by defining a set of tasks and processes ...