This book presents a comprehensive and consistent theory of estimation. The framework described leads naturally to the maximum capacity estimator as a generalization of the maximum likelihood estimator. This approach allows the optimal estimation of real-valued parameters, their number and intervals, as well as providing common ground for explaining the power of these estimators.
Beginning with a review of coding and the key properties of information, the author goes on to discuss the techniques of estimation, and develops the generalized maximum capacity estimator, based on a new form of Shannon’s mutual information and channel capacity. Applications of this powerful technique in hypothesis testing and denoising are described in detail.
Offering an original and thought-provoking perspective on estimation theory, Jorma Rissanen’s book is of interest to graduate students and researchers in the fields of information theory, probability and statistics, econometrics, and finance.
Author(s): Rissanen J.
Publisher: Cambridge University Press
Year: 2012
Language: English
Pages: 171
Tags: Математика;Теория вероятностей и математическая статистика;Математическая статистика;
Rissanen J. Optimal Estimation of Parameters (CUP, 2012)(ISBN 9781107004740)(T)(K)(600dpi)(171p) ......Page 4
Copyright ......Page 5
Contents v ......Page 6
Preface page vii ......Page 8
1 Introduction 1 ......Page 9
Part I Coding and information 9 ......Page 17
2 Basics of coding 11 ......Page 19
2.1 Coding and probability 12 ......Page 20
2.2 Coding and entropy 15 ......Page 23
2.3 Huffman’s code 16 ......Page 24
2.4 Codes as number representations 18 ......Page 26
2.4.1 Arithmetic codes 21 ......Page 29
3.1 Main definitions 24 ......Page 32
3.2.1 Expected mutual information 25 ......Page 33
3.2.2 Maximum estimation information 27 ......Page 35
Part II Estimation 33 ......Page 41
4 Modeling problems 35 ......Page 43
4.1 Models 36 ......Page 44
4.2 General comments on estimation 39 ......Page 47
4.3 Maximum capacity 41 ......Page 49
4.4 Necessary conditions for optimality 47 ......Page 55
4.5 General and complete MDL principles 51 ......Page 59
5.1 Minmax problems 57 ......Page 65
5.2 Consistency 63 ......Page 71
6 Interval estimation 70......Page 78
6.1 Optimum intervals 71 ......Page 79
6.2 Maximum capacity partition 75 ......Page 83
6.3 Error probability 77 ......Page 85
6.3.1 Asymptotic distinguishability 78 ......Page 86
6.3.2 Partition algorithm for k = 2 81 ......Page 89
7 Hypothesis testing 83 ......Page 91
7.1 General plan 84 ......Page 92
7.2 Test statistics 87 ......Page 95
7.3 Characteristic histograms 88 ......Page 96
7.4 Main tests 92 ......Page 100
7.4.1 Simple hypothesis 96 ......Page 104
7.4.2 Composite hypothesis 98 ......Page 106
7.4.3 Likelihood ratio 102 ......Page 110
8 Denoising 104 ......Page 112
8.1 Hard thresholding 105 ......Page 113
8.2 Soft thresholding 109 ......Page 117
9 Sequential models 112 ......Page 120
9.1 Bernoulli class 114 ......Page 122
9.2 Variable order Markov chains 116 ......Page 124
9.2.1 Algorithm Context 119 ......Page 127
9.2.2 Tree pruning 120 ......Page 128
9.2.3 Universal code 123 ......Page 131
9.2.4 Extension to time series 124 ......Page 132
9.3 Linear quadratic regression models 127 ......Page 135
9.3.2 Free variance 129 ......Page 137
9.4 Autoregressive moving average (ARMA) models 133 ......Page 141
9.4.1 Prediction 134 ......Page 142
9.4.2 Prediction bound with estimated parameters 141 ......Page 149
A Elements of algorithmic information 144 ......Page 152
A.l Recursive and partial recursive functions 145 ......Page 153
A. 1.1 Universal computers 147 ......Page 155
A. 1.2 Relative randomness and typicality 149 ......Page 157
A.2 Kolmogorov structure function 150 ......Page 158
B Universal prior for integers 153 ......Page 161
References 156 ......Page 164
Index 161 ......Page 169
cover......Page 1
