How good are humans at solving CAPTCHAs? A large scale evaluation

Elie Bursztein, Steven Bethard, Celine Fabry, John C. Mitchell, Dan Jurafsky

Research output: Chapter in Book/Report/Conference proceedingConference contribution

152 Scopus citations

Abstract

Captchas are designed to be easy for humans but hard for machines. However, most recent research has focused only on making them hard for machines. In this paper, we present what is to the best of our knowledge the first large scale evaluation of captchas from the human perspective, with the goal of assessing how much friction captchas present to the average user. For the purpose of this study we have asked workers from Amazon's Mechanical Turk and an underground captchabreaking service to solve more than 318 000 captchas issued from the 21 most popular captcha schemes (13 images schemes and 8 audio scheme). Analysis of the resulting data reveals that captchas are often difficult for humans, with audio captchas being particularly problematic. We also find some demographic trends indicating, for example, that non-native speakers of English are slower in general and less accurate on English-centric captcha schemes. Evidence from a week's worth of eBay captchas (14,000,000 samples) suggests that the solving accuracies found in our study are close to real-world values, and that improving audio captchas should become a priority, as nearly 1% of all captchas are delivered as audio rather than images. Finally our study also reveals that it is more effective for an attacker to use Mechanical Turk to solve captchas than an underground service.

Original languageEnglish (US)
Title of host publication2010 IEEE Symposium on Security and Privacy, SP 2010 - Proceedings
Pages399-413
Number of pages15
DOIs
StatePublished - 2010
Externally publishedYes
Event31st IEEE Symposium on Security and Privacy, SP 2010 - Berkeley/Oakland, CA, United States
Duration: May 16 2010May 18 2010

Publication series

NameProceedings - IEEE Symposium on Security and Privacy
ISSN (Print)1081-6011

Conference

Conference31st IEEE Symposium on Security and Privacy, SP 2010
Country/TerritoryUnited States
CityBerkeley/Oakland, CA
Period5/16/105/18/10

ASJC Scopus subject areas

  • Safety, Risk, Reliability and Quality
  • Software
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'How good are humans at solving CAPTCHAs? A large scale evaluation'. Together they form a unique fingerprint.

Cite this