The fundamental mathematical tools needed to understand machine learning include linear algebra, analytic geometry, matrix decompositions, vector calculus, optimization, probability and statistics. These topics are traditionally taught in disparate courses, making it hard for data science or computer science students, or professionals, to efficiently learn the mathematics. This self contained textbook bridges the gap between mathematical and machine learning texts, introducing the mathematical concepts with a minimum of prerequisites. It uses these concepts to derive four central machine learning methods: linear regression, principal component analysis, Gaussian mixture models and support vector machines. For students and others with a mathematical background, these derivations provide a starting point to machine learning texts. For those learning the mathematics for the first time, the methods help build intuition and practical experience with applying mathematical concepts. Every chapter includes worked examples and exercises to test understanding. Programming tutorials are offered on the book's web site.
>> https://mml-book.github.io/book/mml-book.pdf (Draft)
https://mml-book.github.io/book/mml-book_printed.pdf
https://github.com/mml-book/mml-book.github.io (Source Code)
https://sml-group.cc/links/
Author(s): Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong
Publisher: Cambridge University Press
Year: 2019
Language: English
Pages: 398
Foreword......Page 7
Part I Mathematical Foundations......Page 15
1 Introduction and Motivation......Page 17
1.1 Finding Words for Intuitions......Page 18
1.2 Two Ways to Read This Book......Page 19
1.3 Exercises and Feedback......Page 22
2 Linear Algebra......Page 23
2.1 Systems of Linear Equations......Page 25
2.2 Matrices......Page 28
2.3 Solving Systems of Linear Equations......Page 33
2.4 Vector Spaces......Page 41
2.5 Linear Independence......Page 46
2.6 Basis and Rank......Page 50
2.7 Linear Mappings......Page 54
2.8 Affine Spaces......Page 67
Exercises......Page 69
3 Analytic Geometry......Page 76
3.1 Norms......Page 77
3.2 Inner Products......Page 78
3.3 Lengths and Distances......Page 81
3.4 Angles and Orthogonality......Page 82
3.5 Orthonormal Basis......Page 84
3.6 Orthogonal Complement......Page 85
3.7 Inner Product of Functions......Page 86
3.8 Orthogonal Projections......Page 87
3.9 Rotations......Page 97
3.10 Further Reading......Page 100
Exercises......Page 101
4 Matrix Decompositions......Page 104
4.1 Determinant and Trace......Page 105
4.2 Eigenvalues and Eigenvectors......Page 111
4.3 Cholesky Decomposition......Page 120
4.4 Eigendecomposition and Diagonalization......Page 121
4.5 Singular Value Decomposition......Page 125
4.6 Matrix Approximation......Page 135
4.7 Matrix Phylogeny......Page 140
4.8 Further Reading......Page 141
Exercises......Page 143
5 Vector Calculus......Page 145
5.1 Differentiation of Univariate Functions......Page 147
5.2 Partial Differentiation and Gradients......Page 152
5.3 Gradients of Vector-Valued Functions......Page 155
5.4 Gradients of Matrices......Page 161
5.5 Useful Identities for Computing Gradients......Page 164
5.6 Backpropagation and Automatic Differentiation......Page 165
5.7 Higher-Order Derivatives......Page 170
5.8 Linearization and Multivariate Taylor Series......Page 171
Exercises......Page 176
6.1 Construction of a Probability Space......Page 178
6.2 Discrete and Continuous Probabilities......Page 184
6.3 Sum Rule, Product Rule, and Bayes' Theorem......Page 189
6.4 Summary Statistics and Independence......Page 192
6.5 Gaussian Distribution......Page 203
6.6 Conjugacy and the Exponential Family......Page 211
6.7 Change of Variables/Inverse Transform......Page 220
6.8 Further Reading......Page 227
Exercises......Page 228
7 Continuous Optimization......Page 231
7.1 Optimization Using Gradient Descent......Page 233
7.2 Constrained Optimization and Lagrange Multipliers......Page 239
7.3 Convex Optimization......Page 242
7.4 Further Reading......Page 252
Exercises......Page 253
Part II Central Machine Learning Problems......Page 255
8.1 Data, Models, and Learning......Page 257
8.2 Empirical Risk Minimization......Page 264
8.3 Parameter Estimation......Page 271
8.4 Probabilistic Modeling and Inference......Page 278
8.5 Directed Graphical Models......Page 284
8.6 Model Selection......Page 289
9 Linear Regression......Page 295
9.1 Problem Formulation......Page 297
9.2 Parameter Estimation......Page 298
9.3 Bayesian Linear Regression......Page 309
9.4 Maximum Likelihood as Orthogonal Projection......Page 319
9.5 Further Reading......Page 321
10 Dimensionality Reduction with Principal Component Analysis......Page 323
10.1 Problem Setting......Page 324
10.2 Maximum Variance Perspective......Page 326
10.3 Projection Perspective......Page 331
10.4 Eigenvector Computation and Low-Rank Approximations......Page 339
10.5 PCA in High Dimensions......Page 341
10.6 Key Steps of PCA in Practice......Page 342
10.7 Latent Variable Perspective......Page 345
10.8 Further Reading......Page 349
11 Density Estimation with Gaussian Mixture Models......Page 354
11.1 Gaussian Mixture Model......Page 355
11.2 Parameter Learning via Maximum Likelihood......Page 356
11.3 EM Algorithm......Page 366
11.4 Latent-Variable Perspective......Page 369
11.5 Further Reading......Page 374
12 Classification with Support Vector Machines......Page 376
12.1 Separating Hyperplanes......Page 378
12.2 Primal Support Vector Machine......Page 380
12.3 Dual Support Vector Machine......Page 389
12.4 Kernels......Page 394
12.5 Numerical Solution......Page 396
12.6 Further Reading......Page 398
References......Page 401
Index......Page 413
