Demographics and Dynamics of Mechanical Turk Workers

Citations Over TimeTop 1% of 2018 papers

Abstract

We present an analysis of the population dynamics and demographics of Amazon Mechanical Turk workers based on the results of the survey that we conducted over a period of 28 months, with more than 85K responses from 40K unique participants. The demographics survey is ongoing (as of November 2017), and the results are available at http://demographics.mturk-tracker.com: we provide an API for researchers to download the survey data. We use techniques from the field of ecology, in particular, the capture-recapture technique, to understand the size and dynamics of the underlying population. We also demonstrate how to model and account for the inherent selection biases in such surveys. Our results indicate that there are more than 100K workers available in Amazon»s crowdsourcing platform, the participation of the workers in the platform follows a heavy-tailed distribution, and at any given time there are more than 2K active workers. We also show that the half-life of a worker on the platform is around 12-18 months and that the rate of arrival of new workers balances the rate of departures, keeping the overall worker population relatively stable. Finally, we demonstrate how we can estimate the biases of different demographics to participate in the survey tasks, and show how to correct such biases. Our methodology is generic and can be applied to any platform where we are interested in understanding the dynamics and demographics of the underlying user population.

Related Papers

• → What? How? Where? A Survey of Crowdsourcing(2013)38 cited
• → Obstacles of Mobile Crowdsourcing: A Survey(2019)8 cited
• → Crowdsourcing as a Future Collaborative Computing Paradigm(2023)1 cited
• → Forms of Crowdsourcing(2013)1 cited
• How to Increase the Accuracy of Crowdsourcing Campaigns(2015)