TY - GEN
T1 - Differentially Private Federated Learning with Drift Control
AU - Chang, Wei Ting
AU - Seif, Mohamed
AU - Tandon, Ravi
N1 - Funding Information:
This work has been supported in part by NSF Grants CAREER 1651492, CNS 1715947, CCF 2100013 and the 2018 Keysight Early Career Professor Award.
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - In this paper, we consider the problem of differentially private federated learning with statistical data heterogeneity. More specifically, users collaborate with the parameter server (PS) to jointly train a machine learning model using their local datasets that are non-i.i.d. across users. The PS is assumed to be honest-but-curious so that the data at users need to be kept private from the PS. More specifically, interactions between the PS and users must satisfy differential privacy (DP) for each user. In this work, we propose a differentially private mechanism that simultaneously deals with user-drift caused by non-i.i.d. data and the randomized user participation in the training process. Specifically, we study SCAFFOLD, a popular federated learning algorithm, that has shown better performance on dealing with non-i.i.d. data than previous federated averaging algorithms. We study the convergence rate of SCAFFOLD under differential privacy constraint. Our convergence results take into account time-varying perturbation noises used by the users, and data and user sampling. We propose two time-varying noise allocation schemes in order to achieve better convergence rate and satisfy a total DP privacy budget. We also conduct experiments to confirm our theoretical findings on real world dataset.
AB - In this paper, we consider the problem of differentially private federated learning with statistical data heterogeneity. More specifically, users collaborate with the parameter server (PS) to jointly train a machine learning model using their local datasets that are non-i.i.d. across users. The PS is assumed to be honest-but-curious so that the data at users need to be kept private from the PS. More specifically, interactions between the PS and users must satisfy differential privacy (DP) for each user. In this work, we propose a differentially private mechanism that simultaneously deals with user-drift caused by non-i.i.d. data and the randomized user participation in the training process. Specifically, we study SCAFFOLD, a popular federated learning algorithm, that has shown better performance on dealing with non-i.i.d. data than previous federated averaging algorithms. We study the convergence rate of SCAFFOLD under differential privacy constraint. Our convergence results take into account time-varying perturbation noises used by the users, and data and user sampling. We propose two time-varying noise allocation schemes in order to achieve better convergence rate and satisfy a total DP privacy budget. We also conduct experiments to confirm our theoretical findings on real world dataset.
KW - Federated learning
KW - Rényi Differential Privacy
KW - Sampling
KW - Stochastic Gradient Descent
UR - http://www.scopus.com/inward/record.url?scp=85128791845&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85128791845&partnerID=8YFLogxK
U2 - 10.1109/CISS53076.2022.9751200
DO - 10.1109/CISS53076.2022.9751200
M3 - Conference contribution
AN - SCOPUS:85128791845
T3 - 2022 56th Annual Conference on Information Sciences and Systems, CISS 2022
SP - 240
EP - 245
BT - 2022 56th Annual Conference on Information Sciences and Systems, CISS 2022
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 56th Annual Conference on Information Sciences and Systems, CISS 2022
Y2 - 9 March 2022 through 11 March 2022
ER -