Model Selection and Multi-Model Inference

0387953647......Page 1
Contents......Page 14
Preface......Page 8
About the Authors......Page 22
Glossary......Page 24
1.1 Objectives of the Book......Page 28
1.2.1 Inference from Data, Given a Model......Page 32
1.2.2 Likelihood and Least Squares Theory......Page 33
1.2.3 The Critical Issue: "What Is the Best Model to Use?"......Page 40
1.2.4 Science Inputs: Formulation of the Set of Candidate Models......Page 42
1.2.5 Models Versus Full Reality......Page 47
1.2.6 An Ideal Approximating Model......Page 49
1.3.2 Approximating Models g[sub(i)](x|θ)......Page 50
1.3.4 Estimated Models [equation omitted]......Page 52
1.3.6 Global Model......Page 53
1.3.7 Overview of Stochastic Models in the Biological Sciences......Page 54
1.4.1 Avoid Overfitting to Achieve a Good Model Fit......Page 56
1.4.2 The Principle of Parsimony......Page 58
1.4.3 Model Selection Methods......Page 62
1.5 Data Dredging, Overanalysis of Data, and Spurious Effects......Page 64
1.5.1 Overanalysis of Data......Page 65
1.5.2 Some Trends......Page 67
1.6 Model Selection Bias......Page 70
1.7 Model Selection Uncertainty......Page 72
1.8 Summary......Page 74
2 Information and Likelihood Theory: A Basis for Model Selection and Inference......Page 76
2.1 Kullback–Leibler Information or Distance Between Two Models......Page 77
2.1.1 Examples of Kullback–Leibler Distance......Page 81
2.1.2 Truth, f, Drops Out as a Constant......Page 85
2.2 Akaike's Information Criterion: 1973......Page 87
2.3 Takeuchi's Information Criterion: 1976......Page 92
2.4 Second-Order Information Criterion: 1978......Page 93
2.5 Modification of Information Criterion for Overdispersed Count Data......Page 94
2.6 AIC Differences, Δ[sub(i)]......Page 97
2.7 A Useful Analogy......Page 99
2.8 Likelihood of a Model, [equation omitted]......Page 101
2.9.1 Basic Formula......Page 102
2.9.2 An Extension......Page 103
2.10 Evidence Ratios......Page 104
2.11.1 AIC Cannot Be Used to Compare Models of Different Data Sets......Page 107
2.11.3 Transformations of the Response Variable......Page 108
2.11.4 Regression Models with Differing Error Structures......Page 109
2.11.6 Null Hypothesis Testing Is Still Important in Strict Experiments......Page 110
2.11.8 Exploratory Data Analysis......Page 111
2.12 Some History and Further Insights......Page 112
2.12.1 Entropy......Page 113
2.12.3 More on Interpreting Information-Theoretic Criteria......Page 114
2.12.4 Nonnested Models......Page 115
2.12.5 Further Insights......Page 116
2.13 Bootstrap Methods and Model Selection Frequencies π[sub(i)]......Page 117
2.13.1 Introduction......Page 118
2.13.2 The Bootstrap in Model Selection: The Basic Idea......Page 120
2.14 Return to Flather's Models......Page 121
2.15 Summary......Page 123
3.1 Introduction......Page 125
3.2 Example 1: Cement Hardening Data......Page 127
3.2.1 Set of Candidate Models......Page 128
3.2.2 Some Results and Comparisons......Page 129
3.3 Example 2: Time Distribution of an Insecticide Added to a Simulated Ecosystem......Page 133
3.3.1 Set of Candidate Models......Page 135
3.3.2 Some Results......Page 137
3.4 Example 3: Nestling Starlings......Page 138
3.4.1 Experimental Scenario......Page 139
3.4.3 Set of Candidate Models......Page 140
3.4.4 Data Analysis Results......Page 144
3.4.5 Further Insights into the First Fourteen Nested Models......Page 147
3.4.6 Hypothesis Testing and Information-Theoretic Approaches Have Different Selection Frequencies......Page 148
3.4.7 Further Insights Following Final Model Selection......Page 151
3.4.8 Why Not Always Use the Global Model for Inference?......Page 152
3.5.1 Introduction......Page 153
3.5.2 Set of Candidate Models......Page 154
3.5.3 Model Selection......Page 156
3.5.4 Hypothesis Tests for Year-Dependent Survival Probabilities......Page 158
3.5.5 Hypothesis Testing Versus AIC in Model Selection......Page 159
3.5.6 A Class of Intermediate Models......Page 161
3.6 Example 5: Resource Utilization of Anolis Lizards......Page 164
3.6.2 Comments on Analytic Method......Page 165
3.6.3 Some Tentative Results......Page 166
3.7 Example 6: Sakamoto et al.'s (1986) Simulated Data......Page 168
3.8 Example 7: Models of Fish Growth......Page 169
3.9 Summary......Page 170
4.1 Introduction to Multimodel Inference......Page 176
4.2.1 Prediction......Page 177
4.2.2 Averaging Across Model Parameters......Page 178
4.3 Model Selection Uncertainty......Page 180
4.3.1 Concepts of Parameter Estimation and Model Selection Uncertainty......Page 182
4.3.2 Including Model Selection Uncertainty in Estimator Sampling Variance......Page 185
4.3.3 Unconditional Confidence Intervals......Page 191
4.4 Estimating the Relative Importance of Variables......Page 194
4.5.1 Introduction......Page 196
4.5.2 Δ[sub(i)], Model Selection Probabilities, and the Bootstrap......Page 198
4.6 Model Redundancy......Page 200
4.7 Recommendations......Page 203
4.8 Cement Data......Page 204
4.9 Pine Wood Data......Page 210
4.10 The Durban Storm Data......Page 214
4.10.1 Models Considered......Page 215
4.10.2 Consideration of Model Fit......Page 217
4.10.3 Confidence Intervals on Predicted Storm Probability......Page 218
4.10.4 Comparisons of Estimator Precision......Page 220
4.11 Flour Beetle Mortality: A Logistic Regression Example......Page 222
4.12 Publication of Research Results......Page 228
4.13 Summary......Page 230
5.1 Introduction......Page 233
5.2.1 A Chain Binomial Survival Model......Page 234
5.2.2 An Example......Page 237
5.2.3 An Extended Survival Model......Page 242
5.2.4 Model Selection if Sample Size Is Huge, or Truth Known......Page 246
5.2.5 A Further Chain Binomial Model......Page 248
5.3 Examples and Ideas Illustrated with Linear Regression......Page 251
5.3.1 All-Subsets Selection: A GPA Example......Page 252
5.3.2 A Monte Carlo Extension of the GPA Example......Page 256
5.3.3 An Improved Set of GPA Prediction Models......Page 262
5.3.4 More Monte Carlo Results......Page 265
5.3.5 Linear Regression and Variable Selection......Page 271
5.3.6 Discussion......Page 275
5.4.1 Density Estimation Background......Page 282
5.4.3 Analysis of Wallaby Creek Data......Page 283
5.4.5 Confidence Interval on D......Page 285
5.4.6 Bootstrap Samples: 1,000 Versus 10,000......Page 287
5.4.7 Bootstrap Versus Akaike Weights: A Lessonon QAIC[sub(c)]......Page 288
5.5 Summary......Page 291
6.1 Introduction......Page 294
6.2.1 Body Fat Data......Page 295
6.2.3 Classical Stepwise Selection......Page 296
6.2.4 Model Selection Uncertainty for AIC[sub(c)] and BIC......Page 298
6.2.5 An A Priori Approach......Page 301
6.2.6 Bootstrap Evaluation of Model Uncertainty......Page 303
6.2.7 Monte Carlo Simulations......Page 306
6.2.8 Summary Messages......Page 308
6.3.1 Criteria That Are Estimates of K-L Information......Page 311
6.3.2 Criteria That Are Consistent for K......Page 313
6.3.3 Contrasts......Page 315
6.3.4 Consistent Selection in Practice: Quasi-true Models......Page 316
6.4.1 A Heuristic Derivation of BIC......Page 320
6.4.2 A K-L-Based Conceptual Comparison of AIC and BIC......Page 322
6.4.3 Performance Comparison......Page 325
6.4.4 Exact Bayesian Model Selection Formulas......Page 328
6.4.5 Akaike Weights as Bayesian Posterior Model Probabilities......Page 329
6.5.1 Overdispersion c and Goodness-of-Fit: A General Strategy......Page 332
6.5.2 Overdispersion Modeling: More Than One c......Page 334
6.5.3 Model Goodness-of-Fit After Selection......Page 336
6.6.1 Basic Concepts and Marginal Likelihood Approach......Page 337
6.6.2 A Shrinkage Approach to AIC and Random Effects......Page 340
6.6.3 On Extensions......Page 343
6.7.1 Keep All the Parts......Page 344
6.7.2 A Normal Versus Log-Normal Example......Page 345
6.7.3 Comparing Across Several Distributions: An Example......Page 347
6.8.1 Use AIC[sub(c)], Not AIC, with Small Sample Sizes......Page 350
6.8.2 Use AIC[sub(c)], Not AIC, When K Is Large......Page 352
6.8.3 When Is AIC[sub(c)] Suitable: A Gamma Distribution Example......Page 353
6.8.4 Inference from a Less Than Best Model......Page 355
6.8.5 Are Parameters Real?......Page 357
6.8.6 Sample Size Is Often Not a Simple Issue......Page 359
6.8.7 Judgment Has a Role......Page 360
6.9.1 Irrelevance of Between-Sample Variation of AIC......Page 361
6.9.2 The G-Statistic and K-L Information......Page 363
6.9.3 AIC Versus Hypothesis Testing: Results Can Be Very Different......Page 364
6.9.4 A Subtle Model Selection Bias Issue......Page 366
6.9.5 The Dimensional Unit of AIC......Page 367
6.9.6 AIC and Finite Mixture Models......Page 369
6.9.7 Unconditional Variance......Page 371
6.9.8 A Baseline for w[sub(+)](i)......Page 372
6.10 Summary......Page 374
7.1 Useful Preliminaries......Page 379
7.2 A General Derivation of AIC......Page 389
7.3.1 Analytical Computation of TIC......Page 398
7.3.2 Bootstrap Estimation of TIC......Page 399
7.4.1 Derivation of AIC[sub(c)]......Page 401
7.4.2 Lack of Uniqueness of AIC[sub(c)]......Page 406
7.5 Derivation of AIC for the Exponential Family of Distributions......Page 407
7.6 Evaluation of [equation omitted] and Its Estimator......Page 411
7.6.1 Comparison of AIC Versus TIC in a Very Simple Setting......Page 412
7.6.2 Evaluation Under Logistic Regression......Page 417
7.6.3 Evaluation Under Multinomially Distributed Count Data......Page 424
7.6.4 Evaluation Under Poisson-Distributed Data......Page 432
7.6.5 Evaluation for Fixed-Effects Normality-Based Linear Models......Page 433
7.7.1 Selection Simulation for Nested Models......Page 439
7.7.2 Simulation of the Distribution of Δ[sub(p)]......Page 442
7.7.3 Does AIC Overfit?......Page 444
7.7.4 Can Selection Be Improved Based on All the Δ[sub(i)]?......Page 446
7.7.5 Linear Regression, AIC, and Mean Square Error......Page 448
7.7.6 AIC[sub(c)] and Models for Multivariate Data......Page 451
7.7.8 Kullback–Leibler Information Relationship to the Fisher Information Matrix......Page 453
7.7.9 Entropy and Jaynes Maxent Principle......Page 454
7.7.10 Akaike Weights w[sub(i)] Versus Selection Probabilities π[sub(i)]......Page 455
7.8 Kullback–Leibler Information Is Always ≥ 0......Page 456
7.9 Summary......Page 461
8 Summary......Page 464
8.1 The Scientific Question and the Collection of Data......Page 466
8.2 Actual Thinking and A Priori Modeling......Page 467
8.3 The Basis for Objective Model Selection......Page 469
8.4 The Principle of Parsimony......Page 470
8.5 Information Criteria as Estimates of Expected Relative Kullback–Leibler Information......Page 471
8.6 Ranking Alternative Models......Page 473
8.7 Scaling Alternative Models......Page 474
8.8 MMI: Inference Based on Model Averaging......Page 475
8.9 MMI: Model Selection Uncertainty......Page 476
8.11 More on Inferences......Page 478
8.12 Final Thoughts......Page 481
References......Page 482
D......Page 512
M......Page 513
S......Page 514
W......Page 515