While there are a number of subjectivity lexicons available for research purposes, none can be used commercially. We describe the process of constructing subjectivity lexicon(s) for recognizing sentiment polarity in essays written by test-takers, to be used within a commercial essay-scoring system. We discuss ways of expanding a manually-built seed lexicon using dictionary-based, distributional...

We consider the problem of task allocation in crowdsourcing systems with multiple complex workflows, each of which consists of a set of inter-dependent micro-tasks. We propose Budgeteer, an algorithm to solve this problem under a budget constraint. In particular, our algorithm first calculates an efficient way to allocate budget to each workflow. It then determines the number of inter-dependent...

Model-finders such as SAT-solvers are attractive for producing concrete models, either as sample instances or as counterexamples when properties fail. However, the generated model is arbitrary. To address this, several research efforts have proposed principled forms of output from model-finders. These include minimal and maximal models, unsat cores, and proof-based provenance of facts. While th...

The prevalent method in theoretical syntax and semantics research involves obtaining a judgment of the acceptability of a sentence ⁄meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. The weakness of the traditional non-quantitative single-sentence ⁄ single-participant methodology, along with the existence of cognitive and social biases, has the unw...

Contemporary visual environments bombard us with hundreds of face images every day, and this places a nontrivial demand on long-term memory. However, little is known about what makes certain faces remain in our memories, while others are quickly forgotten. To establish a basis for face memorability exploration, we assembled a database of 8,690 face photographs from online sources, spanning dive...

This paper considers the linguistic indicators of bias in political text. We used Amazon Mechanical Turk judgments about sentences from American political blogs, asking annotators to indicate whether a sentence showed bias, and if so, in which political direction and through which word tokens. We also asked annotators questions about their own political views. We conducted a preliminary analysi...

Spatial diversity is a relatively new branch of research in the context of spatial information retrieval. Although the assumption that spatially diversified results may meet users’ needs better seems reasonable, there has been little hard evidence in the literature indicating so. In this paper, we will show the potentials of spatial diversity by not only the traditional evaluation metrics (prec...

In studying creativity and techniques for enhancing the creative process, we utilized crowdsourcing through Amazon’s Mechanical Turk. Through our studies we found that in its current form, the Mechanical Turk platform has some limitations in supporting creative tasks. In this position paper, we describe our studies and give recommendations for crowdsourcing platforms that foster creativity.

One major way in which Amazon Mechanical Turk has been used is in the human labeling (or coding) of data, such as the relevance of search results or quality of Wikipedia articles. Recently, we used Amazon Mechanical Turk for classifying or labeling Twitter updates as questions or not. We present the design of our study and the steps that we took to address the challenges we faced in using Mecha...

Word Count: 140 Ms + References Word Count: 1369