A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk)

Introduction Researchers around the globe are utilizing crowdsourcing tools to reach respondents for quantitative and qualitative research (Chambers & Nimon, 2019). Many social science business journals receiving studies that utilize such as Amazon Mechanical Turk (MTurk), Qualtrics, MicroWorkers, ShortTask, ClickWorker, Crowdsource (e.g., Ahn, Back, 2019; Ali et al., 2021; Esfahani, Ozturk, Jeong, Lee, 2017; Zhang 2017). Even though use of these presents a great opportunity sharing large quantities data quickly, some challenges must also be addressed. The purpose this guide is present basic ideas behind survey provide primer best practices will increase their validity reliability. What research? Crowdsourcing describes collection information, opinions, or other types input from number people, typically via internet, which may not receive (financial) compensation (Hargrave, Oxford Dictionary, n.d.). Within behavioral realm, defined internet services hosting activities creating opportunities population participants. Applications techniques have evolved over decades, establishing strong informational power crowds. advent Web 2.0 has expanded possibilities crowdsourcing, with new online reviews, forums, Wikipedia, MTurk, but platforms Crowdflower Prolific Academic (Peer Sheehan, 2018). in age remote labor recruited assist employers complete tasks cannot left machines. Key characteristics include payment workers, recruitment any location, completion (Behrend 2011). They allow relatively quick compared field, participants rewarded an incentive—often financial compensation. only offers participation pool streamlined process study design, participant recruitment, well integrated system (Buhrmester Also, traditional marketing firms, makes it easier detect possible sampling biases (Garrow 2020). Due advantages reduced costs, diversity participants, flexibility, surged popularity researchers. Advantages MTurk one most popular among researchers, allowing Requesters submit Workers (Cummings Sibona, been used platform human subjects purposes (Paolacci Chandler, 2014). Research shown reliable cost-effective tool, capable providing representative sciences Crump 2013; Goodman Mason Suri, 2012; Rand, Simcox Fiez, In addition its studies, marketing, hospitality tourism, psychology, political science, communication, sociology contexts (Sheehan, To illustrate, between 2012 2017, more than 40% published Journal Consumer websites (Goodman Paolacci, Disadvantages Although researchers assessed sciences, they exempt flaws. One disadvantage possibility unsatisfactory quality. fact, virtual setting implies investigator physically separated participant, lack monitoring could lead quality issues addition, on always who claim be, trust provided and, ultimately, findings (McGonagle, 2015; Smith 2016). A recurrent concern instance, assessment experienced takers (Chandler 2015). This experience mainly acquired through dozens surveys per day, especially when faced similar items scales. al. (2016) identified two problems performing using MTurk; namely, cheaters speeders. As Qualtrics—which strict screening quality-control processes ensure be—MTurk appears less exigent regarding workers. However, downside Qualtrics expensive fees—about $5.00 questionnaire against $0.50 $1.50 (Ford, Hence, few were able conduct compare respondent pools firms Another challenge arises trying collect desired responses targeted specific city area (Ross 2010). inherent selection subject investigations several Berinsky Chandler 2014; Harms DeSimone, Paolacci 2010; 2012). Feitosa (2015) pointed out international still identify themselves U.S. fake addresses accounts. found 5% 10% identifying actually overseas locations. Moreover, Babin trap questions allowed uncover many change genders, ages, careers, income within course single survey. (a) workers control (b) speeders, which, can attributed being main source revenue given respondent, remain purposes. Best Some recommended IDs matched previous thus exclude had answered Furthermore, proceed manual assignment qualification prior (Litman Park Park, When dealing both multiple attention checks optimizing way exposed stimuli sufficient length time better address sense, shorter preferred longer ones, affect participant’s concentration, may, turn, adversely impact answers. Most importantly, pretest make sure all parts working expected. should keep mind context primary method measurement web interface. Thus, avoid biases, ponder whether factors emerge latent models (Podsakoff such, time-lagged designs predictor criterion variables measured at different points administered platforms, vs (Cheung general, including appropriate according question; reliant strategies by enhance Trade-offs various need prioritized objectives From our own editorial team members chair conferences, we outlined below: Worker (Respondent) Selection: consider before collection. population. For example, if targets restaurant owners company CEOs, suitable study. target diners, hotel guests, grocery shoppers, students, hourly employees, sample would suitable. tool software. you country, came protocol (IP) outside country report results section. demographics reflect focuses baby boomers technology, then boomers. Similarly, gender balance, racial composition, people mirror problematic response patterns. provides approval rate respondents. refers how times rejected reasons (i.e., wrong code entered). We recommend 90% higher rate. places type your knowledge-based about subject. rather asking “How accounting practices?”, supplemental question “Which following component statement?” into section Survey Validity: pilot fix potential programming entire set collected. Researcher estimate required average calculating incentive equate exceed minimum wage country. build validity-check them ask check “please click ‘strongly agree’ question” “What 2+2? Please choose 5” (Cobanoglu 2016) good implemented, bots easily answer correctly, give random answers questions. Instead, building involved same forms. beginning year birth end effective replying honestly. Exclude those differently. Report methodology. Cavusoglu (2019) almost 20% eliminated due failure embedded forms his aware bot, software runs automated tasks. bot reply surveys. this, Captcha verification, forces perform moving bar certain area, clicking boxes cars, checking verify person taking bot. Whenever appropriate, limit options offered taker spend advance next question. want watch video, read scenario, look picture respond days during week diverse sample. Data Cleaning: do simply select nonsense text. study, manually inspect data. anyone filled too quickly. excluding take takes 10 minutes fill survey, everyone fills 4 less. After groups, speeders’ (aka cheaters) was significantly regular group. needed. Our rule thumb 30% 500 clean wanted, least 650 available after cleaning article, showing editor reviewers taken steps reliability responses. Calculating samples possible. calculate active (Ali 2021). It calculated raw numbers deducted results. 1000 100 coming IP United States, another failing questions, 800/1000= 80%.