Handbook of Bayesian Variable Selection

Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Contents
Preface
Biography
List of Contributors
List of Symbols
I. Spike-and-Slab Priors
1. Discrete Spike-and-Slab Priors: Models and Computational Aspects
1.1. Introduction
1.2. Spike-and-Slab Priors for Linear Regression Models
1.2.1. Stochastic Search MCMC
1.2.2. Prediction via Bayesian Model Averaging
1.3. Spike-and-Slab Priors for Non-Gaussian Data
1.3.1. Compositional Count Data
1.4. Structured Spike-and-Slab Priors for Biomedical Studies
1.4.1. Network Priors
1.4.2. Spiked Nonparametric Priors
1.5. Scalable Bayesian Variable Selection
1.5.1. Variational Inference
1.6. Conclusion
Bibliography
2. Recent Theoretical Advances with the Discrete Spike-and Slab Priors
2.1. Introduction
2.2. Optimal Recovery in Gaussian Sequence Models
2.2.1. Minimax Rate in Nearly Black Gaussian Mean Models
2.2.2. Optimal Bayesian Recovery in `q-norm
2.2.3. Optimal Contraction Rate for Other Variants of Priors
2.2.4. Slow Contraction Rate for Light-tailed Priors
2.3. Sparse Linear Regression Model
2.3.1. Prior Construction and Assumptions
2.3.2. Compatibility Conditions on the Design Matrix
2.3.3. Posterior Contraction Rate
2.3.4. Variable Selection Consistency
2.3.5. Variable Selection with Discrete Spike and Zellner's g-Priors
2.3.6. Bernstein-von Mises Theorem for the Posterior Distribution
2.4. Extension to Generalized Linear Models
2.4.1. Construction of the GLM Family
2.4.2. Clipped GLM and Connections to Regression Settings
2.4.3. Construction of Sparsity Favoring Prior
2.4.4. Assumptions on Data Generating Distribution and Prior
2.4.5. Adaptive Rate-Optimal Posterior Contraction Rate in `1-norm
2.5. Optimality Results for Variational Inference in Linear Regression Models
2.6. Discussion
Bibliography
3. Theoretical and Computational Aspects of Continuous Spike-and-Slab Priors
3.1. Introduction
3.2. Variable Selection in Linear Models
3.3. Continuous Spike-and-Slab Priors
3.3.1. Shrinking and Diffusing Priors
3.3.2. Spike-and-Slab LASSO
3.4. Theoretical Properties
3.4.1. Variable Selection Consistency
3.4.2. Novel Insights
3.4.3. Examples
3.5. Computations
3.5.1. Skinny Gibbs for Scalable Posterior Sampling
3.5.2. Skinny Gibbs for Non-Normal Spike-and-Slab Priors
3.6. Generalizations
3.7. Conclusion
Bibliography
4. Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO
4.1. Introduction
4.2. Variable Selection in High-Dimensions: Frequentist and Bayesian Strategies
4.2.1. Penalized Likelihood Approaches
4.2.2. Spike-and-Slab Priors
4.3. The Spike-and-Slab LASSO
4.3.1. Prior Specification
4.3.2. Selective Shrinkage and Self-Adaptivity to Sparsity
4.3.3. The Spike-and-Slab LASSO in Action
4.4. Computational Details
4.4.1. Coordinate-wise Optimization
4.4.2. Dynamic Posterior Exploration
4.4.3. EM Implementation of the Spike-and-Slab LASSO
4.5. Uncertainty Quanti cation
4.5.1. Debiasing the Posterior Mode
4.5.2. Posterior Sampling for the Spike-and-Slab LASSO
4.6. Illustrations
4.6.1. Example on Synthetic Data
4.6.2. Bardet-Beidl Syndrome Gene Expression Study
4.7. Methodological Extensions
4.8. Theoretical Properties
4.9. Discussion
Bibliography
5. Adaptive Computational Methods for Bayesian Variable Selection
5.1. Introduction
5.1.1. Some Reasons to be Cheerful
5.1.2. Adaptive Monte Carlo Methods
5.2. Some Adaptive Approaches to Bayesian Variable Selection
5.3. Two Adaptive Algorithms
5.3.1. Linear Regression
5.3.2. Non-Gaussian Models
5.4. Examples
5.4.1. Simulated Example: Linear Regression
5.4.2. Fine Mapping for Systemic Lupus Erythematosus
5.4.3. Analysing Environmental DNA Data
5.5. Discussion
Bibliography
II. Continuous Shrinkage Priors
6. Theoretical Guarantees for the Horseshoe and Other Global-Local Shrinkage Priors
6.1. Introduction
6.1.1. Model and Notation
6.1.2. Global-Local Shrinkage Priors and Spike-and-Slab Priors
6.1.3. Performance Measures
6.2. Global-Local Shrinkage Priors
6.3. Recovery Guarantees
6.3.1. Non-Adaptive Posterior Concentration Theorems
6.3.2. Proof Techniques
6.3.3. Adaptive Posterior Concentration Theorems
6.3.4. Other Sparsity Assumptions
6.3.5. Implications for Practice
6.4. Uncertainty Quanti cation Guarantees
6.4.1. Credible Intervals
6.4.2. Credible Balls
6.4.3. Implications for Practice
6.5. Variable Selection Guarantees
6.5.1. Thresholding on the Amount of Shrinkage
6.5.2. Checking for Zero in Marginal Credible Intervals
6.6. Discussion
Bibliography
7. MCMC for Global-Local Shrinkage Priors in High-Dimensional Settings
7.1. Introduction
7.2. Global-Local Shrinkage Priors
7.3. Posterior Sampling
7.3.1. Sampling Structured High-Dimensional Gaussians
7.3.2. Blocking can be Advantageous
7.3.3. Geometric Convergence
7.4. Approximate MCMC
7.5. Conclusion
Bibliography
8. Variable Selection with Shrinkage Priors via Sparse Posterior Summaries
8.1. Introduction
8.2. Penalized Credible Region Selection
8.2.1. Gaussian Prior
8.2.2. Global-Local Shrinkage Priors
8.2.3. Example: Simulation Studies
8.2.4. Example: Mouse Gene Expression Real-time PCR
8.3. Approaches Based on Other Posterior Summaries
8.4. Model Selection for Logistic Regression
8.5. Graphical Model Selection
8.6. Confounder Selection
8.7. Time-Varying Coefficients
8.8. Discussion
Bibliography
III. Extensions to Various Modeling Frameworks
9. Bayesian Model Averaging in Causal Inference
9.1. Introduction to Causal Inference
9.1.1. Potential Outcomes, Estimands, and Identifying Assumptions
9.1.2. Estimation Strategies Using Outcome Regression, Propensity Scores, or Both
9.1.3. Why Use BMA for Causal Inference?
9.2. Failure of Traditional Model Averaging for Causal Inference Problems
9.3. Prior Distributions Tailored Towards Causal Estimation
9.3.1. Bayesian Adjustment for Confounding Prior
9.3.2. Related Prior Distributions that Link Treatment and Outcome Models
9.4. Bayesian Estimation of Treatment Effects
9.4.1. Outcome Model Based Estimation
9.4.2. Incorporating the Propensity Score into the Outcome Model
9.4.3. BMA Coupled with Traditional Frequentist Estimators
9.4.4. Analysis of Volatile Compounds on Cholesterol Levels
9.5. Assessment of Uncertainty
9.6. Extensions to Shrinkage Priors and Nonlinear Regression
9.7. Conclusion
Bibliography
10. Variable Selection for Hierarchically-Related Outcomes: Models and Algorithms
10.1. Introduction
10.2. Model Formulations, Computational Challenges and Tradeoffs
10.3. Illustrations on Published Case Studies
10.3.1. Modelling eQTL Signals across Multiple Tissues
10.3.2. Modelling eQTL Hotspots under Different Experimental Conditions
10.4. Discussion
Bibliography
11. Bayesian Variable Selection in Spatial Regression Models
11.1. Introduction
11.2. Spatial Regression
11.3. Regression Coefficients as Spatial Processes
11.3.1. Spatially-Varying Coe cient Model
11.3.2. Scalar-on-Image Regression
11.4. Sparse Spatial Processes
11.4.1. Discrete Mixture Priors
11.4.2. Continuous Shrinkage Priors
11.5. Application to Microbial Fungi across US Households
11.6. Discussion
Bibliography
12. Effect Selection and Regularization in Structured Additive Distributional Regression
12.1. Introduction
12.2. Structured Additive Distributional Regression
12.2.1. Basic Model Structure
12.2.2. Predictor Components
12.2.3. Common Response Distributions
12.2.4. Basic MCMC Algorithm
12.3. Effect Selection Priors
12.3.1. Challenges
12.3.2. Spike-and-Slab Priors for Effect Selection
12.3.3. Regularization Priors for Effect Selection
12.4. Application: Childhood Undernutrition in India
12.4.1. Data
12.4.2. A Main Effects Location-Scale Model
12.4.3. Decomposing an Interaction Surface
12.5. Other Regularization Priors for Functional Effects
12.5.1. Locally Adaptive Regularization
12.5.2. Shrinkage towards a Functional Subspace
12.6. Summary and Discussion
Bibliography
13. Sparse Bayesian State-Space and Time-Varying Parameter Models
13.1. Introduction
13.2. Univariate Time-Varying Parameter Models
13.2.1. Motivation and Model Definition
13.2.2. The Inverse Gamma Versus the Ridge Prior
13.2.3. Gibbs Sampling in the Non-Centered Parametrization
13.3. Continuous Shrinkage Priors for Sparse TVP Models
13.3.1. From the Ridge Prior to Continuous Shrinkage Priors
13.3.2. Efficient MCMC Inference
13.3.3. Application to US Inflation Modelling
13.4. Spike-and-Slab Priors for Sparse TVP Models
13.4.1. From the Ridge prior to Spike-and-Slab Priors
13.4.2. Model Space MCMC
13.4.3. Application to US Inflation Modelling
13.5. Extensions
13.5.1. Including Stochastic Volatility
13.5.2. Sparse TVP Models for Multivariate Time Series
13.5.3. Non-Gaussian Outcomes
13.5.4. Log Predictive Scores for Comparing Shrinkage Priors
13.5.5. BMA Versus Continuous Shrinkage Priors
13.6. Discussion
Bibliography
14. Bayesian Estimation of Single and Multiple Graphs
14.1. Introduction
14.2. Bayesian Approaches for Single Graph Estimation
14.2.1. Background on Graphical Models
14.2.2. Bayesian Priors for Undirected Networks
14.2.3. Bayesian Priors for Directed Networks
14.2.4. Bayesian Network Inference for Non-Gaussian Data
14.3. Multiple Graphs with Shared Structure
14.3.1. Likelihood
14.3.2. Prior Formulation
14.3.3. Simulation and Case Studies
14.3.4. Related Work
14.4. Multiple Graphs with Shared Edge Values
14.4.1. Likelihood
14.4.2. Prior Formulation
14.4.3. Analysis of Neuroimaging Data
14.5. Multiple DAGs and Other Multiple Graph Approaches
14.6. Related Topics
14.7. Discussion
Bibliography
IV. Other Approaches to Bayesian Variable Selection
15. Bayes Factors Based on g-Priors for Variable Selection
15.1. Bayes Factors
15.2. Variable Selection in the Gaussian Linear Model
15.2.1. Objective Prior Specifications
15.2.2. Numerical Issues
15.2.3. BayesVarSel and Applications
15.2.4. Sensitivity to Prior Inputs
15.3. Variable Selection for Non-Gaussian Data
15.3.1. glmBfp and Applications
15.4. Conclusion
Bibliography
16. Balancing Sparsity and Power: Likelihoods, Priors, and Misspecification
16.1. Introduction
16.2. BMS in Regression Models
16.3. Interpreting BMS Under Misspeci cation
16.4. Priors
16.5. Prior Elicitation and Robustness
16.6. Validity of Model Selection Uncertainty
16.7. Finite-Dimensional Results
16.8. High-Dimensional Results
16.9. Balancing Sparsity and Power
16.10. Examples
16.10.1. Salary
16.10.2. Colon Cancer
16.10.3. Survival Analysis of Serum Free Light Chain Data
16.11. Discussion
Bibliography
17. Variable Selection and Interaction Detection with Bayesian Additive Regression Trees
17.1. Introduction
17.2. BART Overview
17.2.1. Specification of the BART Regularization Prior
17.2.2. Posterior Calculation and Information Extraction
17.3. Model-Free Variable Selection with BART
17.3.1. Variable Selection with the Boston Housing Data
17.4. Model-Free Interaction Detection with BART
17.4.1. Variable Selection and Interaction Detection with the Friedman Simulation Setup
17.4.2. Interaction Detection with the Boston Housing Data
17.5. A Utility Based Approach to Variable Selection using BART Inference
17.5.1. Step 1: BART Inference
17.5.2. Step 2: Subset Search
17.5.3. Step 3: Uncertainty Assessment
17.6. Conclusion
Bibliography
18. Variable Selection for Bayesian Decision Tree Ensembles
18.1. Introduction
18.1.1. Running Example
18.1.2. Possible Strategies
18.2. Bayesian Additive Regression Trees
18.2.1. Decision Trees and their Priors
18.2.2. The BART Model
18.3. Variable Importance Scores
18.3.1. Empirical Bayes and Variable Importance Scores
18.4. Sparsity Inducing Priors on s
18.4.1. The Uniform Prior on s
18.4.2. The Dirichlet Prior
18.4.3. The Spike-and-Forest Prior
18.4.4. Finite Gibbs Priors
18.5. An Illustration: The WIPP Dataset
18.6. Extensions
18.6.1. Interaction Detection
18.6.2. Structure in Predictors
18.7. Discussion
Bibliography
19. Stochastic Partitioning for Variable Selection in Multivariate Mixture of Regression Models
19.1. Introduction
19.2. Mixture of Univariate Regression Models
19.2.1. Model Fitting
19.2.2. Variable Selection
19.3. Stochastic Partitioning for Multivariate Mixtures
19.3.1. Model Formulation
19.3.2. Prior Speci cation
19.3.3. Model Fitting
19.3.4. Posterior Inference
19.4. spavs and Application
19.4.1. Choice of Hyperparameters and Other Input Values
19.4.2. Post-Processing of MCMC Output and Posterior Inference
19.5. Discussion
Bibliography
Index