LASSO - simultaneous shrinkage and selection via the L1 norm

Abstract

Two major purposes of regression models are explanation and prediction of scientific phenomena. Explanation is obtained by producing interpretable models through variable selection, while prediction accuracy is optimised by balancing the bias and variance of predictions. This dissertation explores the LASSO, a shrinkage method that simultaneously performs selection and estimation, yielding interpretable models with high prediction accuracy. By penalizing the regression model, the variance is substantially reduced and sparsity is promoted by using the L1 norm. It often outperforms traditional methods like subset selection and ridge regression, each focusing either on variable selection or prediction, respectively. The LASSO has favourable statistical properties and can also be applied to high dimensional data. Applied in two-stage procedures, the bias is controlled to achieve consistency for both prediction and selection. Concave penalties reduce the bias more effectively by applying different penalty functions over fixed ranges of each coefficient’s size. Adaptations of the LASSO penalty allow incorporating different structures between predictors, such as ordering predictors in a meaningful way or including known groups of predictors like dummy variables or polynomials. Penalties combining the L1 norm with other norms allow the identification of unknown groups of correlated variables. Overall the LASSO provides an elegant foundation for a class of methods which improves the way that sparse regression problems are solved.