TY - JOUR
T1 - How Algorithms Discriminate Based on Data They Lack Challenges, Solutions, and Policy Implications
AU - Williams, Betsy Anne
AU - Brooks, Catherine F.
AU - Shmargad, Yotam
N1 - Publisher Copyright:
© 2018 Penn State University Press. All rights reserved.
PY - 2018/3
Y1 - 2018/3
N2 - Organizations often employ data-driven models to inform decisions that can have a significant impact on people’s lives (e.g., university admissions, hiring). In order to protect people’s privacy and prevent discrimination, these decision-makers may choose to delete or avoid collecting social category data, like sex and race. In this article, we argue that such censoring can exacerbate discrimination by making biases more difficult to detect. We begin by detailing how computerized decisions can lead to biases in the absence of social category data and in some contexts, may even sustain biases that arise by random chance. We then show how proactively using social category data can help illuminate and combat discriminatory practices, using cases from education and employment that lead to strategies for detecting and preventing discrimination. We conclude that discrimination can occur in any sociotechnical system in which someone decides to use an algorithmic process to inform decision-making, and we offer a set of broader implications for researchers and policymakers.
AB - Organizations often employ data-driven models to inform decisions that can have a significant impact on people’s lives (e.g., university admissions, hiring). In order to protect people’s privacy and prevent discrimination, these decision-makers may choose to delete or avoid collecting social category data, like sex and race. In this article, we argue that such censoring can exacerbate discrimination by making biases more difficult to detect. We begin by detailing how computerized decisions can lead to biases in the absence of social category data and in some contexts, may even sustain biases that arise by random chance. We then show how proactively using social category data can help illuminate and combat discriminatory practices, using cases from education and employment that lead to strategies for detecting and preventing discrimination. We conclude that discrimination can occur in any sociotechnical system in which someone decides to use an algorithmic process to inform decision-making, and we offer a set of broader implications for researchers and policymakers.
KW - Algorithmic discrimination
KW - Omitted variables
KW - Personal data
KW - Signaling
KW - Statistical discrimination
UR - http://www.scopus.com/inward/record.url?scp=85114802853&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85114802853&partnerID=8YFLogxK
U2 - 10.5325/JINFOPOLI.8.1.0078
DO - 10.5325/JINFOPOLI.8.1.0078
M3 - Article
AN - SCOPUS:85114802853
SN - 2381-5892
VL - 8
SP - 78
EP - 115
JO - Journal of Information Policy
JF - Journal of Information Policy
IS - 1
ER -