Matrix Algebra from a Statistician's Perspective

Title......Page 1
Preface......Page 6
Contents......Page 12
1.1 Basic Terminology......Page 20
1.2 Basic Operations......Page 21
1.3 Some Basic Types of Matrices......Page 25
Exercises......Page 29
2.1 Some Terminology and Basic Results......Page 32
2.2 Scalar Multiples, Transposes, Sums, and Products of Partitioned Matrices......Page 36
2.3 Some Results on the Product of a Matrix and a Column Vector......Page 38
2.4 Expansion of a Matrix in Terms of Its Rows, Columns, or Elements......Page 39
Exercises......Page 41
3.1 Definitions......Page 42
3.2 Some Basic Results......Page 43
Exercises......Page 45
4.1 Some Definitions, Notation, and BasicRelationships and Properties......Page 46
4.2 Subspaces......Page 48
4.3 Bases......Page 50
4.4 Rank of a Matrix......Page 55
4.5 Some Basic Results on Partitioned Matrices and on Sums of Matrices......Page 59
Exercises......Page 65
5.1 Definition and Basic Properties......Page 68
5.2 Trace of a Product......Page 69
5.3 Some Equivalent Conditions......Page 71
Exercises......Page 72
6.1 Definitions: Norm, Distance, Angle, Inner Product, and Orthogonality......Page 74
6.2 Orthogonal and Orthonormal Sets......Page 80
6.3 Schwarz Inequality......Page 81
6.4 Orthonormal Bases......Page 82
Exercises......Page 87
7.1 Some Basic Terminology......Page 90
7.2 Consistency......Page 91
7.3 Compatibility......Page 92
7.4 Linear Systems of the Form A'AX = A'B......Page 93
Exercise......Page 96
8.1 Some Definitions and Basic Results......Page 98
8.2 Properties of Inverse Matrices......Page 100
8.3 Premultiplication or Postmultiplication by a Matrix of Full Column or Row Rank......Page 102
8.4 Orthogonal Matrices......Page 103
8.5 Some Basic Results on the Ranks and Inverses of Partitioned Matrices......Page 107
Exercises......Page 122
9.1 Definition, Existence, and a Connection to the Solution of Linear Systems......Page 126
9.2 Some Alternative Characterizations......Page 128
9.3 Some Elementary Properties......Page 136
9.4 Invariance to the Choice of a Generalized Inverse......Page 138
9.5 A Necessary and Sufficient Condition for the Consistency of a Linear System......Page 139
9.6 Some Results on the Ranks and Generalized Inverses of Partitioned Matrices......Page 140
9.7 Extension of Some Results on Systems of the Form AX = B to Systems of the Form AXC = B......Page 143
Exercises......Page 144
10.1 Definition and Some Basic Properties......Page 151
10.2 Some Basic Results......Page 152
Exercises......Page 154
11.1 Some Terminology, Notation, and Basic Results......Page 157
11.2 General Form of a Solution......Page 158
11.3 Number of Solutions......Page 160
11.4 A Basic Result on Null Spaces......Page 161
11.5 An Alternative Expression for the General Form of a Solution......Page 162
11.6 Equivalent Linear Systems......Page 163
11.8 Linear Systems With Nonsingular Triangular or Block-Triangular Coefficient Matrices......Page 164
11.9 A Computational Approach......Page 167
11.10 Linear Combinations of the Unknowns......Page 168
11.11 Absorption......Page 170
11.12 Extensions to Systems of the Form AXC = B......Page 175
Exercises......Page 176
12.1 Some General Results, Terminology, and Notation......Page 179
12.2 Projection of a Column Vector......Page 181
12.3 Projection Matrices......Page 184
12.4 Least Squares Problem......Page 187
12.5 Orthogonal Complements......Page 189
Exercises......Page 193
13.1 Some Definitions, Notation, and Special Cases......Page 195
13.2 Some Basic Properties of Determinants......Page 199
13.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices......Page 203
13.5 Cofactors......Page 207
13.6 Vandermonde Matrices......Page 211
13.7 Some Results on the Determinant of the Sum of Two Matrices......Page 213
13.8 Laplace’s Theorem and the Binet-Cauchy Formula......Page 215
Exercises......Page 220
14.1 Some Terminology and Basic Results......Page 225
14.2 Nonnegative Definite Quadratic Forms and Matrices......Page 228
14.3 Decomposition of Symmetric and Symmetric Nonnegative Definite Matrices......Page 233
14.4 Generalized Inverses of Symmetric Nonnegative Definite Matrices......Page 238
14.5 LDU, U'DU, and Cholesky Decompositions......Page 239
14.6 Skew-Symmetric Matrices......Page 255
14.7 Trace of a Nonnegative Definite Matrix......Page 256
14.8 Partitioned Nonnegative Definite Matrices......Page 258
14.9 Some Results on Determinants......Page 263
14.10 Geometrical Considerations......Page 270
14.11 Some Results on Ranks and Row and Column Spaces and on Linear Systems......Page 274
14.12 Projections, Projection Matrices, and Orthogonal Complements......Page 275
Exercises......Page 291
15 Matrix Differentiation......Page 303
15.1 Definitions, Notation, and Other Preliminaries......Page 304
15.2 Differentiation of (Scalar-Valued) Functions: Some Elementary Results......Page 310
15.3 Differentiation of Linear and Quadratic Forms......Page 312
15.4 Differentiation of Matrix Sums, Products, and Transposes (and of Matrices of Constants)......Page 314
15.5 Differentiation of a Vector or an (Unrestricted or Symmetric) Matrix With Respect to Its Elements......Page 317
15.6 Differentiation of a Trace of a Matrix......Page 318
15.7 The Chain Rule......Page 320
15.8 First-Order Partial Derivatives of Determinants and Inverse and Adjoint Matrices......Page 322
15.9 Second-Order Partial Derivatives of Determinants and Inverse Matrices......Page 326
15.10 Differentiation of Generalized Inverses......Page 327
15.11 Differentiation of Projection Matrices......Page 332
15.12 Evaluation of Some Multiple Integrals......Page 338
Exercises......Page 341
Bibliographic and Supplementary Notes......Page 349
16.1 The Kronecker Product of Two or More Matrices: Definition and Some Basic Properties......Page 351
16.2 The Vec Operator: Definition and Some Basic Properties......Page 357
16.3 Vec-Permutation Matrix......Page 361
16.4 The Vech Operator......Page 368
16.5 Reformulation of a Linear System......Page 381
16.6 Some Results on Jacobian Matrices......Page 383
Exercises......Page 386
Bibliographic and Supplementary Notes......Page 392
17.1 Definitions and Some Basic Properties......Page 395
17.2 Some Results on Row and Column Spaces and on the Ranks of Partitioned Matrices......Page 401
17.3 Some Results on Linear Systems and on Generalized Inverses of Partitioned Matrices......Page 408
17.4 Subspaces: Sum of Their Dimensions Versus Dimension of Their Sum......Page 411
17.5 Some Results on the Rank of a Product of Matrices......Page 414
17.6 Projections Along a Subspace......Page 417
17.7 Some Further Results on the Essential Disjointness and Orthogonality of Subspaces and on Projections and Projection Matrices......Page 424
Exercises......Page 426
Bibliographic and Supplementary Notes......Page 432
18 Sums (and Differences) of Matrices......Page 433
18.1 Some Results on Determinants......Page 434
18.2 Some Results on Inverses and Generalized Inverses and on Linear Systems......Page 437
18.3 Some Results on Positive (and Nonnegative) Definiteness......Page 450
18.4 Some Results on Idempotency......Page 452
18.5 Some Results on Ranks......Page 458
Exercises......Page 463
Bibliographic and Supplementary Notes......Page 471
19 Minimization of a Second-Degree Polynomial (in n Variables) Subject to Linear Constraints......Page 473
19.1 Unconstrained Minimization......Page 474
19.2 Constrained Minimization......Page 477
19.3 Explicit Expressions for Solutions to the Constrained Minimization Problem......Page 482
19.4 Some Results on Generalized Inverses of Partitioned Matrices......Page 491
19.5 Some Additional Results on the Form of Solutions to the Constrained Minimization Problem......Page 497
19.6 Transformation of the Constrained Minimization Problem to an Unconstrained Minimization Problem......Page 503
19.7 The Effect of Constraints on the Generalized Least Squares Problem......Page 504
Exercises......Page 506
Bibliographic and Supplementary Notes......Page 509
20.1 Definition, Existence, and Uniqueness (of the Moore-Penrose Inverse)......Page 511
20.2 Some Special Cases......Page 513
20.3 Special Types of Generalized Inverses......Page 514
20.4 Some Alternative Representations and Characterizations......Page 520
20.5 Some Basic Properties and Relationships......Page 522
20.6 Minimum Norm Solution to the Least Squares Problem......Page 525
20.7 Expression of the Moore-Penrose Inverse as a Limit......Page 526
20.8 Differentiation of the Moore-Penrose Inverse......Page 528
Exercises......Page 531
Bibliographic and Supplementary Notes......Page 532
21 Eigenvalues and Eigenvectors......Page 533
21.1 Definitions, Terminology, and Some Basic Results......Page 534
21.2 Eigenvalues of Triangular or Block-TriangularMatrices and of Diagonal or Block-DiagonalMatrices......Page 540
21.3 Similar Matrices......Page 542
21.4 Linear Independence of Eigenvectors......Page 546
21.5 Diagonalization......Page 549
21.6 Expressions for the Trace and Determinant of a Matrix......Page 557
21.7 Some Results on the Moore-Penrose Inverse of a Symmetric Matrix......Page 558
21.8 Eigenvalues of Orthogonal, Idempotent, and Nonnegative Definite Matrices......Page 559
21.9 Square Root of a Symmetric Nonnegative DefiniteMatrix......Page 561
21.10 Some Relationships......Page 563
21.11 Eigenvalues and Eigenvectors of Kronecker Products of (Square) Matrices......Page 565
21.12 Singular Value Decomposition......Page 568
21.13 Simultaneous Diagonalization......Page 577
21.14 Generalized Eigenvalue Problem......Page 580
21.15 Differentiation of Eigenvalues and Eigenvectors......Page 582
21.16 An Equivalence (Involving Determinants and Polynomials)......Page 585
Appendix: Some Properties of Polynomials (in a Single Variable)......Page 591
Exercises......Page 593
Bibliographic and Supplementary Notes......Page 599
22.1 Some Definitions, Terminology, and Basic Results......Page 601
22.2 Scalar Multiples, Sums, and Products of Linear Transformations......Page 607
22.3 Inverse Transformations and Isomorphic Linear Spaces......Page 610
22.4 Matrix Representation of a Linear Transformation......Page 613
22.5 Terminology and Properties Shared by a Linear Transformation and Its Matrix Representation......Page 621
22.6 Linear Functionals and Dual Transformations......Page 624
Exercises......Page 627
References......Page 633
Index......Page 637