Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks
Citations Over TimeTop 10% of 2017 papers
Abstract
Paid crowdsourcing platforms have evolved into remarkable marketplaces where requesters can tap into human intelligence to serve a multitude of purposes, and the workforce can benefit through monetary returns for investing their efforts. In this work, we focus on individual crowd worker competencies. By drawing from self-assessment theories in psychology, we show that crowd workers often lack awareness about their true level of competence. Due to this, although workers intend to maintain a high reputation, they tend to participate in tasks that are beyond their competence. We reveal the diversity of individual worker competencies, and make a case for competence-based pre-selection in crowdsourcing marketplaces. We show the implications of flawed self-assessments on real-world microtasks, and propose a novel worker pre-selection method that considers accuracy of worker self-assessments. We evaluated our method in a sentiment analysis task and observed an improvement in the accuracy by over 15%, when compared to traditional performance-based worker pre-selection. Similarly, our proposed method resulted in an improvement in accuracy of nearly 6% in an image validation task. Our results show that requesters in crowdsourcing platforms can benefit by considering worker self-assessments in addition to their performance for pre-selection.
Related Papers
- → What? How? Where? A Survey of Crowdsourcing(2013)38 cited
- → Obstacles of Mobile Crowdsourcing: A Survey(2019)8 cited
- → Crowdsourcing as a Future Collaborative Computing Paradigm(2023)1 cited
- → Forms of Crowdsourcing(2013)1 cited
- How to Increase the Accuracy of Crowdsourcing Campaigns(2015)