Science consists in using information about the world for the purpose of predicting, explaining, understanding, and/or controlling phenomena of interest.
The basic dfficulty is that the available information is usually insufficient to attain any of those goals with certainty. A central concern in these lectures will be the problem of inductive inference, that is, the problem of reasoning under conditions of incomplete information.
Our goal is twofold. First, to develop the main tools for inference - probability and entropy - and to demonstrate their use. And second, to demonstrate their importance for physics. More specifically our goal is to clarify the conceptual foundations of physics by deriving the fundamental laws of statistical mechanics and of quantum mechanics as examples of inductive inference. Perhaps all physics can be derived in this way.
Author(s): Caticha, Ariel
Publisher: International Society for Bayesian Analysis
Year: 2012
Language: English
Pages: 293
Tags: Физика;Матметоды и моделирование в физике;
Foreword iii
Preface 1
1 Inductive Inference and Physics 1
1.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Designing a framework for inductive inference . . . . . . . . . . . 4
1.3 Entropic Physics . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Probability 7
2.1 The design of probability theory . . . . . . . . . . . . . . . . . . 8
2.1.1 Rational beliefs? . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 Quantifying rational belief . . . . . . . . . . . . . . . . . . 9
2.2 The sum rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 The associativity constraint . . . . . . . . . . . . . . . . . 13
2.2.2 The general solution and its regraduation . . . . . . . . . 14
2.2.3 The general sum rule . . . . . . . . . . . . . . . . . . . . . 15
2.2.4 Cox's proof . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 The product rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.1 From four arguments down to two . . . . . . . . . . . . . 18
2.3.2 The distributivity constraint . . . . . . . . . . . . . . . . 20
2.4 Some remarks on the sum and product rules . . . . . . . . . . . . 22
2.4.1 On meaning, ignorance and randomness . . . . . . . . . . 22
2.4.2 Independent and mutually exclusive events . . . . . . . . 23
2.4.3 Marginalization . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5 The expected value . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.6 The binomial distribution . . . . . . . . . . . . . . . . . . . . . . 26
2.7 Probability vs. frequency: the law of large numbers . . . . . . . . 28
2.8 The Gaussian distribution . . . . . . . . . . . . . . . . . . . . . . 30
2.8.1 The de Moivre-Laplace theorem . . . . . . . . . . . . . . 30
2.8.2 The Central Limit Theorem . . . . . . . . . . . . . . . . . 33
2.9 Updating probabilities: Bayes' rule . . . . . . . . . . . . . . . . . 35
2.9.1 Formulating the problem . . . . . . . . . . . . . . . . . . 35
2.9.2 Minimal updating: Bayes' rule . . . . . . . . . . . . . . . 36
2.9.3 Multiple experiments, sequential updating . . . . . . . . . 40
2.9.4 Remarks on priors . . . . . . . . . . . . . . . . . . . . . . 41
2.10 Hypothesis testing and con rmation . . . . . . . . . . . . . . . . 44
2.11 Examples from data analysis . . . . . . . . . . . . . . . . . . . . 48
2.11.1 Parameter estimation . . . . . . . . . . . . . . . . . . . . 48
2.11.2 Curve tting . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.11.3 Model selection . . . . . . . . . . . . . . . . . . . . . . . . 53
2.11.4 Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . 54
3 Entropy I: The Evolution of Carnot's Principle 57
3.1 Carnot: reversible engines . . . . . . . . . . . . . . . . . . . . . . 57
3.2 Kelvin: temperature . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.3 Clausius: entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.4 Maxwell: probability . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.5 Gibbs: beyond heat . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.6 Boltzmann: entropy and probability . . . . . . . . . . . . . . . . 67
3.7 Some remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4 Entropy II: Measuring Information 73
4.1 Shannon's information measure . . . . . . . . . . . . . . . . . . . 74
4.2 Relative entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3 Joint entropy, additivity, and subadditivity . . . . . . . . . . . . 82
4.4 Conditional entropy and mutual information . . . . . . . . . . . 83
4.5 Continuous distributions . . . . . . . . . . . . . . . . . . . . . . . 84
4.6 Experimental design . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.7 Communication Theory . . . . . . . . . . . . . . . . . . . . . . . 89
4.8 Assigning probabilities: MaxEnt . . . . . . . . . . . . . . . . . . 92
4.9 Canonical distributions . . . . . . . . . . . . . . . . . . . . . . . . 93
4.10 On constraints and relevant information . . . . . . . . . . . . . . 96
4.11 Avoiding pitfalls { I . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.11.1 MaxEnt cannot x
awed information . . . . . . . . . . . 99
4.11.2 MaxEnt cannot supply missing information . . . . . . . . 100
4.11.3 Sample averages are not expected values . . . . . . . . . . 100
5 Statistical Mechanics 103
5.1 Liouville's theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.2 Derivation of Equal a Priori Probabilities . . . . . . . . . . . . . 105
5.3 The relevant constraints . . . . . . . . . . . . . . . . . . . . . . . 108
5.4 The canonical formalism . . . . . . . . . . . . . . . . . . . . . . . 110
5.5 Equilibrium with a heat bath of nite size . . . . . . . . . . . . . 113
5.6 The Second Law of Thermodynamics . . . . . . . . . . . . . . . . 115
5.7 The thermodynamic limit . . . . . . . . . . . . . . . . . . . . . . 118
5.8 Interpretation of the Second Law: Reproducibility . . . . . . . . 122
5.9 Remarks on irreversibility . . . . . . . . . . . . . . . . . . . . . . 123
5.10 Entropies, descriptions and the Gibbs paradox . . . . . . . . . . 125
6 Entropy III: Updating Probabilities 131
6.1 What is information? . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.2 The design of entropic inference . . . . . . . . . . . . . . . . . . . 137
6.2.1 General criteria . . . . . . . . . . . . . . . . . . . . . . . 138
6.2.2 Entropy as a tool for updating probabilities . . . . . . . . 140
6.2.3 Speci c design criteria . . . . . . . . . . . . . . . . . . . . 141
6.2.4 The ME method . . . . . . . . . . . . . . . . . . . . . . . 145
6.3 The proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4 An alternative independence criterion: consistency . . . . . . . . 152
6.5 Random remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.5.1 On priors . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.5.2 Comments on other axiomatizations . . . . . . . . . . . . 162
6.6 Bayes' rule as a special case of ME . . . . . . . . . . . . . . . . . 163
6.7 Commuting and non-commuting constraints . . . . . . . . . . . . 168
6.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7 Information Geometry 173
7.1 Examples of statistical manifolds . . . . . . . . . . . . . . . . . . 174
7.2 Vectors in curved spaces . . . . . . . . . . . . . . . . . . . . . . . 175
7.3 Distance and volume in curved spaces . . . . . . . . . . . . . . . 178
7.4 Derivations of the information metric . . . . . . . . . . . . . . . . 180
7.4.1 Derivation from distinguishability . . . . . . . . . . . . . 180
7.4.2 Derivation from a Euclidean metric . . . . . . . . . . . . . 181
7.4.3 Derivation from asymptotic inference . . . . . . . . . . . . 182
7.4.4 Derivation from relative entropy . . . . . . . . . . . . . . 185
7.5 Uniqueness of the information metric . . . . . . . . . . . . . . . . 185
7.6 The metric for some common distributions . . . . . . . . . . . . . 194
8 Entropy IV: Entropic Inference 199
8.1 Deviations from maximum entropy . . . . . . . . . . . . . . . . . 199
8.2 The ME method . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.3 An application to
uctuations . . . . . . . . . . . . . . . . . . . . 202
8.4 Avoiding pitfalls { II . . . . . . . . . . . . . . . . . . . . . . . . . 205
8.4.1 The three-sided die . . . . . . . . . . . . . . . . . . . . . . 206
8.4.2 Understanding ignorance . . . . . . . . . . . . . . . . . . 208
9 Entropic Dynamics: Time and Quantum Theory 213
9.1 The statistical model . . . . . . . . . . . . . . . . . . . . . . . . . 216
9.2 Entropic dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . 218
9.3 Entropic time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.3.1 Time as a sequence of instants . . . . . . . . . . . . . . . 221
9.4 Duration: a convenient time scale . . . . . . . . . . . . . . . . . . 222
9.4.1 The directionality of entropic time . . . . . . . . . . . . . 223
9.5 Accumulating changes . . . . . . . . . . . . . . . . . . . . . . . . 225
9.5.1 Derivation of the Fokker-Planck equation . . . . . . . . . 226
9.5.2 The current and osmotic velocities . . . . . . . . . . . . . 227
9.6 Non-dissipative di usion . . . . . . . . . . . . . . . . . . . . . . . 228
9.6.1 Manifold dynamics . . . . . . . . . . . . . . . . . . . . . . 230
9.6.2 Classical limits . . . . . . . . . . . . . . . . . . . . . . . . 232
9.6.3 The Schrodinger equation . . . . . . . . . . . . . . . . . . 234
9.7 A quantum equivalence principle . . . . . . . . . . . . . . . . . . 235
9.8 Entropic time vs. physical time . . . . . . . . . . . . . . . . . . . 237
9.9 Dynamics in an external electromagnetic eld . . . . . . . . . . . 238
9.9.1 An additional constraint . . . . . . . . . . . . . . . . . . . 238
9.9.2 Entropic dynamics . . . . . . . . . . . . . . . . . . . . . . 238
9.9.3 Gauge invariance . . . . . . . . . . . . . . . . . . . . . . . 240
9.10 Is ED a hidden-variable model? . . . . . . . . . . . . . . . . . . . 242
9.11 Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . 245
10 Topics in Quantum Theory 249
10.1 The quantum measurement problem . . . . . . . . . . . . . . . . 249
10.2 Observables other than position . . . . . . . . . . . . . . . . . . . 252
10.3 Ampli cation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
10.4 But isn't the measuring device a quantum system too? . . . . . . 256
10.5 Momentum in Entropic Dynamics . . . . . . . . . . . . . . . . . 258
10.5.1 Expected values . . . . . . . . . . . . . . . . . . . . . . . 260
10.5.2 Uncertainty relations . . . . . . . . . . . . . . . . . . . . . 260
10.5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
10.5.4 An aside: the hybrid = 0 theory . . . . . . . . . . . . . 264
10.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
References 267
