“If it didn’t happen, why would I change my decision?”: How Judges Respond to Counterfactual Explanations for the Public Safety Assessment

Yaniv Yacoby, Ben Green, Christopher L. Griffin, Finale Doshi-Velez

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many researchers and policymakers have expressed excitement about algorithmic explanations enabling more fair and responsible decision-making. However, recent experimental studies have found that explanations do not always improve human use of algorithmic advice. In this study, we shed light on how people interpret and respond to counterfactual explanations(CFEs)—explanations that show how a model’s output would change with marginal changes to its input(s)—in the context of pretrial risk assessment instruments (PRAIs).We ran think-aloud trials with eight sitting U.S. state court judges, providing them with recommendations from a PRAI that includes CFEs. We found that the CFEs did not alter the judges’ decisions. At first, judges misinterpreted the counter factualsas real—rather than hypothetical—changes to defendants. Once judges understood what the counter factualsmeant, they ignored them, stating their role is only to make decisions regarding the actual defendant in question. The judges also expressed a mix of reasons for ignoring or following the advice of the PRAI without CFEs. These results add to the literature detailing the unexpected ways in which people respond to algorithms and explanations. They also highlight new challenges associated with improving human-algorithm collaborations through explanations.

Original languageEnglish (US)
Title of host publicationHCOMP 2022 - Proceedings of the 10th AAAI Conference on Human Computation and Crowdsourcing
EditorsJane Hsu, Ming Yin
PublisherAssociation for the Advancement of Artificial Intelligence
Pages219-230
Number of pages12
ISBN (Print)9781577358787
DOIs
StatePublished - 2022
Event10th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2022 - Virtual, Online
Duration: Nov 6 2022Nov 10 2022

Publication series

NameProceedings of the AAAI Conference on Human Computation and Crowdsourcing
Volume10
ISSN (Print)2769-1330
ISSN (Electronic)2769-1349

Conference

Conference10th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2022
CityVirtual, Online
Period11/6/2211/10/22

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of '“If it didn’t happen, why would I change my decision?”: How Judges Respond to Counterfactual Explanations for the Public Safety Assessment'. Together they form a unique fingerprint.

Cite this