Manual annotators are often used to label human interaction data. This is associated with high costs and high time consumption, which is one reason annotation by crowd sourcing is increasing. But in crowd sourcing, control over the conditions is largely lost. To get increased throughput with a higher measure of experimental control, we suggest borrowing from the Audience Response Systems used e...

We present an evaluation of a Web-deployed spoken language CALL system, carried out using crowd-sourcing methods. The system, “Survival Japanese”, is a crash course in tourist Japanese implemented within the platform CALL-SLT. The evaluation was carried out over one week using the Amazon Mechanical Turk. Although we found a high proportion of attempted scammers, there was a core of 23 subjects ...

In the present study of heterogeneous crowd sourcing is a novel field which shall bring marked improvement in the field of decision support system. At this point of time, the world is evolving technologically and thus we have started emphasizing on human brains as a specific physical sensors as far sensing capabilities are concerned. There are many human beings who are actually Subject Matters ...

We propose the use of a game with a purpose (GWAP) to facilitate crowd-sourcing of phraseequivalents, as an alternative to expert or paid crowd-sourcing. Doodling is an online multiplayer game, in which one player (drawer), draws pictures on a shared board to get the other players (guessers) to guess the meaning behind an assigned phrase. In this paper we describe the system and results from se...

The logistics of collecting resources for Machine Translation (MT) has always been a cause of concern for some of the resource deprived languages of the world. The recent advent of crowdsourcing platforms provides an opportunity to explore the large scale generation of resources for MT. However, before venturing into this mode of resource collection, it is important to understand the various fa...

Crowd-sourcing has become a popular means of acquiring labeled data for many tasks where humans are more accurate than computers, such as image tagging, entity resolution, and sentiment analysis. However, due to the time and cost of human labor, solutions that rely solely on crowd-sourcing are oŸen limited to small datasets (i.e., a few thousand items). is paper proposes algorithms for integrat...

Web service annotation is of crucial importance to enable efficient search and discovery of such services, especially for those that are REST based. However, the process of annotating such services has several drawbacks, whether automated or not. Automated processes are quite inaccurate and do not in general provide quality annotations. Manual processes are very expensive, both in time, effort ...

Crowd-sourcing has become a popular means of acquiring labeled data for many tasks where humans are more accurate than computers, such as image tagging, entity resolution, or sentiment analysis. However, due to the time and cost of human labor, solutions that solely rely on crowd-sourcing are often limited to small datasets (i.e., a few thousand items). This paper proposes algorithms for integr...