Extending the reach of the lasso and elastic net penalties : methodology and practice
Abstract/Contents
- Abstract
- The lasso and its successor, the elastic net, are two popular regularized regression methods which produce sparse models. We propose two different extensions to these methods that leverage feature information in the model-fitting process to improve predictive accuracy. The first method, the "principal components lasso" ("pcLasso"), combines the lasso penalty with a quadratic penalty that shrinks the coefficient vector toward leading principal components of each group. pcLasso is compatible with both overlapping and non-overlapping groups. We provide some theory for the method and illustrate the method on some simulated and real data examples. Next, we develop a general framework for organizing feature information into an auxiliary matrix and propose a method, called the "feature-weighted elastic net" ("fwelnet"), that uses it to adapt the relative penalties on the feature coefficients in the elastic net penalty. We present connections between this method and the group lasso, and also to Bayesian estimation. We then switch gears and explore how to relax another limitation of the lasso and the elastic net: the fact that the model's prediction must be a sparse linear combination of the input features. Motivated by reluctant interaction modeling, we propose a multi-stage algorithm, called "reluctant generalized additive modeling" (RGAM), that can fit sparse generalized additive models (GAMs) at scale. While the individual features are allowed to vary non-linearly with the response, the model remains additive in the input features, thus preserving interpretability. RGAM has an explicit bias toward simpler, linear relationships, with non-linearities only included if they improve predictive power. Unlike existing methods for sparse GAMs, RGAM can be extended easily to binary, count and survival data. Finally, we turn to the practical matter of computation. Earlier versions of the glmnet package, which implements the lasso and the elastic net in R, had specialized FORTRAN subroutines for fitting the models for popular model families, including the logistic regression and Poisson regression models. We describe how the elastic net penalty can be applied to any generalized linear model (GLM), and explain the computational approach needed to make model-fitting feasible in practice. We also show how this approach extends to fitting regularized Cox proportional hazards models for (start, stop] data and for stratified data.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2021; ©2021 |
| Publication date | 2021; 2021 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Tay, Jingyi Kenneth |
|---|---|
| Degree supervisor | Tibshirani, Robert |
| Thesis advisor | Tibshirani, Robert |
| Thesis advisor | Owen, Art B |
| Thesis advisor | Walther, Guenther |
| Degree committee member | Owen, Art B |
| Degree committee member | Walther, Guenther |
| Associated with | Stanford University, Department of Statistics |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Jingyi Kenneth Tay. |
|---|---|
| Note | Submitted to the Department of Statistics. |
| Thesis | Thesis Ph.D. Stanford University 2021. |
| Location | https://purl.stanford.edu/yf024jm4134 |
Access conditions
- Copyright
- © 2021 by Jingyi Kenneth Tay
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...