Beyond the Turk: Alternative platforms for crowdsourcing behavioral research

Eyal Peer, Laura Brandimarte, Sonam Samat, Alessandro Acquisti

Research output: Contribution to journalArticlepeer-review

859 Scopus citations

Abstract

The success of Amazon Mechanical Turk (MTurk) as an online research platform has come at a price: MTurk has suffered from slowing rates of population replenishment, and growing participant non-naivety. Recently, a number of alternative platforms have emerged, offering capabilities similar to MTurk but providing access to new and more naïve populations. After surveying several options, we empirically examined two such platforms, CrowdFlower (CF) and Prolific Academic (ProA). In two studies, we found that participants on both platforms were more naïve and less dishonest compared to MTurk participants. Across the three platforms, CF provided the best response rate, but CF participants failed more attention-check questions and did not reproduce known effects replicated on ProA and MTurk. Moreover, ProA participants produced data quality that was higher than CF's and comparable to MTurk's. ProA and CF participants were also much more diverse than participants from MTurk.

Original languageEnglish (US)
Pages (from-to)153-163
Number of pages11
JournalJournal of Experimental Social Psychology
Volume70
DOIs
StatePublished - May 1 2017
Externally publishedYes

Keywords

  • Amazon Mechanical Turk
  • CrowdFlower
  • Crowdsourcing
  • Data quality
  • Online research
  • Prolific Academic

ASJC Scopus subject areas

  • Social Psychology
  • Sociology and Political Science

Fingerprint

Dive into the research topics of 'Beyond the Turk: Alternative platforms for crowdsourcing behavioral research'. Together they form a unique fingerprint.

Cite this