A Survey of Solution Path Algorithms for Regression and Classification Models

Guangrui Tang, Neng Fan

Research output: Contribution to journalReview articlepeer-review

1 Scopus citations


The loss function expresses the differences between the predicted values from regression or classification models and the actual instances in machine learning. Regularization also plays an important role in machine learning, and it can mitigate overfitting problems, perform variable selection, and produce sparse models. The hyperparameter in these models controls the trade-off between the loss function and the regularization term, as well as the bias-variance trade-off. The choice of hyperparameter will influence the performance of the models. Thus, the hyperparameter needs to be tuned for effective learning from data. In some machine learning models, the optimal values for estimated coefficients are piecewise linear with respect to the hyperparameter. Efficient algorithms can be developed to compute all solutions, and these kinds of methods are called solution path algorithms. They can significantly reduce the efforts for cross-validation and highly speed up hyperparameter tuning. In this paper, we review the solution path algorithms widely used in regression and classification machine learning problems.

Original languageEnglish (US)
Pages (from-to)749-789
Number of pages41
JournalAnnals of Data Science
Issue number4
StatePublished - Aug 2022
Externally publishedYes


  • Classification
  • Machine learning
  • Regression
  • Regularization
  • Solution path

ASJC Scopus subject areas

  • Business, Management and Accounting (miscellaneous)
  • Computer Science Applications
  • Statistics, Probability and Uncertainty
  • Artificial Intelligence

Cite this