TY - GEN
T1 - Sanity Checks for Lottery Tickets
T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
AU - Ma, Xiaolong
AU - Yuan, Geng
AU - Shen, Xuan
AU - Chen, Tianlong
AU - Chen, Xuxi
AU - Chen, Xiaohan
AU - Liu, Ning
AU - Qin, Minghai
AU - Liu, Sijia
AU - Wang, Zhangyang
AU - Wang, Yanzhi
N1 - Publisher Copyright:
© 2021 Neural information processing systems foundation. All rights reserved.
PY - 2021
Y1 - 2021
N2 - There have been long-standing controversies and inconsistencies over the experiment setup and criteria for identifying the “winning ticket” in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with comprehensive and more rigorous conditions. Under our new definition, we show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. Through extensive experiments, we perform quantitative analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well as the architecture characteristics such as capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. Our codes are publicly available at: https://github.com/boone891214/sanity-check-LTH.
AB - There have been long-standing controversies and inconsistencies over the experiment setup and criteria for identifying the “winning ticket” in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with comprehensive and more rigorous conditions. Under our new definition, we show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. Through extensive experiments, we perform quantitative analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well as the architecture characteristics such as capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. Our codes are publicly available at: https://github.com/boone891214/sanity-check-LTH.
UR - https://www.scopus.com/pages/publications/85131920167
UR - https://www.scopus.com/pages/publications/85131920167#tab=citedBy
M3 - Conference contribution
AN - SCOPUS:85131920167
T3 - Advances in Neural Information Processing Systems
SP - 12749
EP - 12760
BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
A2 - Ranzato, Marc'Aurelio
A2 - Beygelzimer, Alina
A2 - Dauphin, Yann
A2 - Liang, Percy S.
A2 - Wortman Vaughan, Jenn
PB - Neural information processing systems foundation
Y2 - 6 December 2021 through 14 December 2021
ER -