Bayesian Modeling and Computation in Python

Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Contents
Foreword
Preface
Symbols
1. Bayesian Inference
1.1. Bayesian Modeling
1.1.1. Bayesian Models
1.1.2. Bayesian Inference
1.2. A DIY Sampler, Do Not Try This at Home
1.3. Say Yes to Automating Inference, Say No to Automated Model Building
1.4. A Few Options to Quantify Your Prior Information
1.4.1. Conjugate Priors
1.4.2. Objective Priors
1.4.3. Maximum Entropy Priors
1.4.4. Weakly Informative Priors and Regularization Priors
1.4.5. Using Prior Predictive Distributions to Assess Priors
1.5. Exercises
2. Exploratory Analysis of Bayesian Models
2.1. There is Life After Inference, and Before Too!
2.2. Understanding Your Assumptions
2.3. Understanding Your Predictions
2.4. Diagnosing Numerical Inference
2.4.1. Effective Sample Size
2.4.2. Potential Scale Reduction Factor R
2.4.3. Monte Carlo Standard Error
2.4.4. Trace Plots
2.4.5. Autocorrelation Plots
2.4.6. Rank Plots
2.4.7. Divergences
2.4.8. Sampler Parameters and Other Diagnostics
2.5. Model Comparison
2.5.1. Cross-validation and LOO
2.5.2. Expected Log Predictive Density
2.5.3. Pareto Shape Parameter, k is Large
2.5.4. Interpreting p_loo When Pareto k
2.5.5. LOO-PIT
2.5.6. Model Averaging
2.6. Exercises
3. Linear Models and Probabilistic Programming Languages
3.1. Comparing Two (or More) Groups
3.1.1. Comparing Two PPLs
3.2. Linear Regression
3.2.1. Linear Penguins
3.2.2. Predictions
3.2.3. Centering
3.3. Multiple Linear Regression
3.3.1. Counterfactuals
3.4. Generalized Linear Models
3.4.1. Logistic Regression
3.4.2. Classifying Penguins
3.4.3. Interpreting Log Odds
3.5. Picking Priors in Regression Models
3.6. Exercises
4. Extending Linear Models
4.1. Transforming Covariates
4.2. Varying Uncertainty
4.3. Interaction Effects
4.4. Robust Regression
4.5. Pooling, Multilevel Models, and Mixed Effects
4.5.1. Unpooled Parameters
4.5.2. Pooled Parameters
4.5.3. Mixing Group and Common Parameters
4.6. Hierarchical Models
4.6.1. Posterior Geometry Matters
4.6.2. Predictions at Multiple Levels
4.6.3. Priors for Multilevel Models
4.7. Exercises
5. Splines
5.1. Polynomial Regression
5.2. Expanding the Feature Space
5.3. Introducing Splines
5.4. Building the Design Matrix using Patsy
5.5. Fitting Splines in PyMC3
5.6. Choosing Knots and Prior for Splines
5.6.1. Regularizing Prior for Splines
5.7. Modeling CO2 Uptake with Splines
5.8. Exercises
6. Time Series
6.1. An Overview of Time Series Problems
6.2. Time Series Analysis as a Regression Problem
6.2.1. Design Matrices for Time Series
6.2.2. Basis Functions and Generalized Additive Model
6.3. Autoregressive Models
6.3.1. Latent AR Process and Smoothing
6.3.2. (S)AR(I)MA(X)
6.4. State Space Models
6.4.1. Linear Gaussian State Space Models and Kalman filter
6.4.2. ARIMA, Expressed as a State Space Model
6.4.3. Bayesian Structural Time Series
6.5. Other Time Series Models
6.6. Model Criticism and Choosing Priors
6.6.1. Priors for Time Series Models
6.7. Exercises
7. Bayesian Additive Regression Trees
7.1. Decision Trees
7.1.1. Ensembles of Decision Trees
7.2. The BART Model
7.3. Priors for BART
7.3.1. Prior Independence
7.3.2. Prior for the Tree Structure Tj
7.3.3. Prior for the Leaf Values uij and Number of Trees m
7.4. Fitting Bayesian Additive Regression Trees
7.5. BART Bikes
7.6. Generalized BART Models
7.7. Interpretability of BARTs
7.7.1. Partial Dependence Plots
7.7.2. Individual Conditional Expectation
7.8. Variable Selection
7.9. Priors for BART in PyMC3
7.10. Exercises
8. Approximate Bayesian Computation
8.1. Life Beyond Likelihood
8.2. Approximating the Approximated Posterior
8.3. Fitting a Gaussian the ABC-way
8.4. Choosing the Distance Function,  and the Summary Statistics
8.4.1. Choosing the Distance
8.4.2. Choosing e
8.4.3. Choosing Summary Statistics
8.5. g-and-k Distribution
8.6. Approximating Moving Averages
8.7. Model Comparison in the ABC Context
8.7.1. Marginal Likelihood and LOO
8.7.2. Model Choice via Random Forest
8.7.3. Model Choice for MA Model
8.8. Choosing Priors for ABC
8.9. Exercises
9. End to End Bayesian Workflows
9.1. Workflows, Contexts, and Questions
9.1.1. Applied Example: Airlines Flight Delays Problem
9.2. Getting Data
9.2.1. Sample Surveys
9.2.2. Experimental Design
9.2.3. Observational Studies
9.2.4. Missing Data
9.2.5. Applied Example: Collecting Airline Flight Delays Data
9.3. Making a Model and Probably More Than One
9.3.1. Questions to Ask Before Building a Bayesian Model
9.3.2. Applied Example: Picking Flight Delay Likelihoods
9.4. Choosing Priors and Predictive Priors
9.4.1. Applied Example: Picking Priors for Flight Delays Model
9.5. Inference and Inference Diagnostics
9.5.1. Applied Example: Running Inference on Flight Delays Models
9.6. Posterior Plots
9.6.1. Applied Example: Posterior of Flight Delays Models
9.7. Evaluating Posterior Predictive Distributions
9.7.1. Applied Example: Posterior Predictive Distributions of Flight Delays
9.8. Model Comparison
9.8.1. Applied Example: Model Comparison with LOO of Flight Delays
9.9. Reward Functions and Decisions
9.9.1. Applied Example: Making Decisions Based on Flight Delays Modeling Results
9.10. Sharing the Results With a Particular Audience
9.10.1. Reproducibility of Analysis Workflow
9.10.2. Understanding the Audience
9.10.3. Static Visual Aids
9.10.4. Reproducible Computing Environments
9.10.5. Applied Example: Presenting Flight Delay Conclusions
9.11. Experimental Example: Comparing Between Two Groups
9.12. Exercises
10. Probabilistic Programming Languages
10.1. A Systems Engineering Perspective of a PPL
10.1.1. Example: Rainier
10.2. Posterior Computation
10.2.1. Getting the Gradient
10.2.2. Example: Near Real Time Inference
10.3. Application Programming Interfaces
10.3.1. Example: Stan and Slicstan
10.3.2. Example: PyMC3 and PyMC4
10.4. PPL Driven Transformations
10.4.1. Log Probabilities
10.4.2. Random Variables and Distributions Transformations
10.4.3. Example: Sampling Comparison between Bounded and Unbounded Random Variables
10.5. Operation Graphs and Automatic Reparameterization
10.6. Effect handling
10.6.1. Example: Effect Handling in TFP and Numpyro
10.7. Base Language, Code Ecosystem, Modularity and Everything Else
10.8. Designing a PPL
10.8.1. Shape Handling in PPLs
10.9. Takeaways for the Applied Bayesian Practitioner
10.10. Exercises
11. Appendiceal Topics
11.1. Probability Background
11.1.1. Probability
11.1.2. Conditional Probability
11.1.3. Probability Distribution
11.1.4. Discrete Random Variables and Distributions
11.1.5. Continuous Random Variables and Distributions
11.1.6. Joint, Conditional and Marginal Distributions
11.1.7. Probability Integral Transform (PIT)
11.1.8. Expectations
11.1.9. Transformations
11.1.10. Limits
11.1.11. Markov Chains
11.2. Entropy
11.3. Kullback-Leibler Divergence
11.4. Information Criterion
11.5. LOO in Depth
11.6. Jeffreys' Prior Derivation
11.6.1. Jeffreys' Prior for the Binomial Likelihood in Terms of θ
11.6.2. Jeffreys' Prior for the Binomial Likelihood in Terms of k
11.6.3. Jeffreys' Posterior for the Binomial Likelihood
11.7. Marginal Likelihood
11.7.1. The Harmonic Mean Estimator
11.7.2. Marginal Likelihood and Model Comparison
11.7.3. Bayes Factor vs WAIC and LOO
11.8. Moving out of Flatland
11.9. Inference Methods
11.9.1. Grid Method
11.9.2. Metropolis-Hastings
11.9.3. Hamiltonian Monte Carlo
11.9.4. Sequential Monte Carlo
11.9.5. Variational Inference
11.10. Programming References
11.10.1. Which Programming Language?
11.10.2. Version Control
11.10.3. Dependency Management and Package Repositories
11.10.4. Environment Management
11.10.5. Text Editor vs Integrated Development Environment vs Notebook
11.10.6. The Specific Tools Used for this Book
Glossary
Bibliography
Index