We conduct the first natural field experiment to explore the relationship between the “meaningfulness” of a task and worker effort. We employed about 2,500 workers from Amazon’s Mechanical Turk (MTurk), an online labor market, to label medical images. Although given an identical task, we experimentally manipulated how the task was framed. Subjects in the meaningful treatment were told that they...

We have conducted three empirical studies of the effects of friend recommendations and general ratings on how online users make choices. These two components of social influence were investigated through user studies on Mechanical Turk. We find that for a user deciding between two choices an additional rating star has a much larger effect than an additional friend’s recommendation on the probab...

How accurate are listeners at identifying the ethnicities of political figures from one-word samples? Do suprasegmental variables provide a basis for these judgments? Tokens of six lexical items were extracted from speeches by seven male political figures of different stated ethnic identities. In a Mechanical Turk experiment, 94 listeners heard each token twice, then responded to the multiplech...

Evidence is accumulating that many temporal features of second language speech are correlated with those of first language speech. This study looks at the correlation between articulation rate, pause rate, and mean pause duration in Japanese first and English second language speech and how second language fluency raters perceive these. In a crosslinguistic corpus of spontaneous speech, mean pau...

Companies are encouraging and incentivizing contributors of online word-of-mouth (WOM) through gamification elements such as badges, mayorships, points, and such. We study how gamification elements, which capture and signal contributors’ accumulated expertise, affect consumers’ perception of contributors’ knowledge, and therefore the perceived effectiveness of their contributed WOM. We focus on...

Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks — making it necessary to understand their behavior over these long periods of time. We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that work...

Formative assessments allow learners to quickly identify knowledge gaps. In traditional educational settings, expert instructors can create assessments, but in informal learning environment, it is difficult for novice learners to self assess because they don’t know what they don’t know. This paper introduces Questimator, an automated system that generates multiple-choice assessment questions fo...

In developing regions, the reach of crowdsourcing services such as Amazon Mechanical Turk (mTurk) has been limited by the lack of adequate payment mechanisms and low visibility amongst the crowd. In this paper, we present a commodity based model for crowdsourcing where crowd workers get paid in kind in the form of a commodity instead of money. Our model makes crowdsourcing services more visible...