Matrix Algebra: Theory, Computations and Applications in Statistics

Preface to Third Edition
Preface to Second Edition
Preface to First Edition
1 Introduction
1.1 Vectors
1.2 Arrays
1.3 Matrices
1.4 Representation of Data
1.5 What You Compute and What You Don't
1.6 R
1.6.1 R Data Types
1.6.2 Program Control Statements
1.6.3 Packages
1.6.4 User Functions and Operators
1.6.5 Generating Artificial Data
1.6.6 Graphics Functions
1.6.7 Special Data in R
1.6.8 Determining Properties of a Computer System in R
1.6.9 Documentation: Finding R Functions and Packages
1.6.10 Documentation: R Functions, Packages, and Other Objects
1.6.11 The Design of the R Programming Language
1.6.12 Why I Use R in This Book
Part I Linear Algebra
2 Vectors and Vector Spaces
2.1 Operations on Vectors
2.1.1 Linear Combinations and Linear Independence
2.1.2 Vector Spaces and Spaces of Vectors
Generating Sets
The Order and the Dimension of a Vector Space
Vector Spaces with an Infinite Number of Dimensions
Some Special Vectors: Notation
Ordinal Relations Among Vectors
Set Operations on Vector Spaces
Essentially Disjoint Vector Spaces
Subpaces
Intersections of Vector Spaces
Unions and Direct Sums of Vector Spaces
Direct Sum Decomposition of a Vector Space
Direct Products of Vector Spaces and Dimension Reduction
2.1.3 Basis Sets for Vector Spaces
Properties of Basis Sets of Vector Subspaces
2.1.4 Inner Products
Inner Products in General Real Vector Spaces
The Inner Product in Real Vector Spaces
2.1.5 Norms
Convexity
Norms Induced by Inner Products
Lp Norms
Basis Norms
Equivalence of Norms
2.1.6 Normalized Vectors
``Inverse'' of a Vector
2.1.7 Metrics and Distances
Metrics Induced by Norms
Convergence of Sequences of Vectors
2.1.8 Orthogonal Vectors and Orthogonal Vector Spaces
2.1.9 The ``One Vector''
The Mean and the Mean Vector
2.2 Cartesian Coordinates and Geometrical Properties of Vectors
2.2.1 Cartesian Geometry
2.2.2 Projections
2.2.3 Angles Between Vectors
2.2.4 Orthogonalization Transformations: Gram–Schmidt
2.2.5 Orthonormal Basis Sets
2.2.6 Approximation of Vectors
Optimality of the Fourier Coefficients
Choice of the Best Basis Subset
2.2.7 Flats, Affine Spaces, and Hyperplanes
2.2.8 Cones
Convex Cones
Dual Cones
Polar Cones
Additional Properties
2.2.9 Vector Cross Products in IR3
2.3 Centered Vectors, and Variances and Covariances of Vectors
2.3.1 The Mean and Centered Vectors
2.3.2 The Standard Deviation, the Variance, and Scaled Vectors
2.3.3 Covariances and Correlations Between Vectors
Appendix: Vectors in R
Exercises
3 Basic Properties of Matrices
3.1 Basic Definitions and Notation
3.1.1 Multiplication of a Matrix by a Scalar
3.1.2 Symmetric and Hermitian Matrices
3.1.3 Diagonal Elements
3.1.4 Diagonally Dominant Matrices
3.1.5 Diagonal and Hollow Matrices
3.1.6 Matrices with Other Special Patterns of Zeroes
3.1.7 Matrix Shaping Operators
Transpose
Conjugate Transpose
Properties
Diagonals of Matrices and Diagonal Vectors: vecdiag(·) or diag(·)
The Diagonal Matrix Constructor Function diag(·)
Forming a Vector from the Elements of a Matrix: vec(·) and vech(·)
3.1.8 Partitioned Matrices and Submatrices
Notation
Block Diagonal Matrices
The diag(·) Matrix Function and the Direct Sum
Transposes of Partitioned Matrices
3.1.9 Matrix Addition
The Transpose of the Sum of Matrices
Rank Ordering Matrices
Vector Spaces of Matrices
3.1.10 The Trace of a Square Matrix
The Trace: tr(·)
The Trace of the Transpose of Square Matrices
The Trace of Scalar Products of Square Matrices
The Trace of Partitioned Square Matrices
The Trace of the Sum of Square Matrices
3.2 The Determinant
3.2.1 Definition and Simple Properties
Notation
3.2.2 Determinants of Various Types of Square Matrices
The Determinant of the Transpose of Square Matrices
The Determinant of Scalar Products of Square Matrices
The Determinant of a Triangular Matrix, Upper or Lower
The Determinant of Square Block Triangular Matrices
The Determinant of the Sum of Square Matrices
3.2.3 Minors, Cofactors, and Adjugate Matrices
An Expansion of the Determinant
Definitions of Terms: Minors and Cofactors
Adjugate Matrices
Cofactors and Orthogonal Vectors
A Diagonal Expansion of the Determinant
3.2.4 A Geometrical Perspective of the Determinant
Computing the Determinant
3.3 Multiplication of Matrices and Multiplication of Vectors and Matrices
3.3.1 Matrix Multiplication (Cayley)
Factorization of Matrices
Powers of Square Matrices
Idempotent and Nilpotent Matrices
Matrix Polynomials
3.3.2 Cayley Multiplication of Matrices with Special Patterns
Multiplication of Matrices and Vectors
The Matrix/Vector Product as a Linear Combination
The Matrix as a Mapping on Vector Spaces
Multiplication of Partitioned Matrices
3.3.3 Elementary Operations on Matrices
Interchange of Rows or Columns: Permutation Matrices
The Vec-Permutation Matrix
Scalar Row or Column Multiplication
Axpy Row or Column Transformations
Gaussian Elimination: Gaussian Transformation Matrix
Elementary Operator Matrices: Summary of Notation and Properties
Determinants of Elementary Operator Matrices
3.3.4 The Trace of a Cayley Product that Is Square
3.3.5 The Determinant of a Cayley Product of Square Matrices
The Adjugate in Matrix Multiplication
3.3.6 Outer Products of Vectors
3.3.7 Bilinear and Quadratic Forms: Definiteness
Nonnegative Definite and Positive Definite Matrices
Ordinal Relations among Symmetric Matrices
The Trace of Inner and Outer Products
3.3.8 Anisometric Spaces
Conjugacy
3.3.9 The Hadamard Product
3.3.10 The Kronecker Product
3.3.11 The Inner Product of Matrices
Orthonormal Matrices
Orthonormal Basis: Fourier Expansion
3.4 Matrix Rank and the Inverse of a Matrix
3.4.1 Row Rank and Column Rank
3.4.2 Full Rank Matrices
3.4.3 Rank of Elementary Operator Matrices and Matrix Products Involving Them
3.4.4 The Rank of Partitioned Matrices, Products of Matrices, and Sums of Matrices
Rank of Partitioned Matrices and Submatrices
An Upper Bound on the Rank of Products of Matrices
An Upper and a Lower Bound on the Rank of Sums of Matrices
3.4.5 Full Rank Partitioning
3.4.6 Full Rank Matrices and Matrix Inverses
Solutions of Linear Equations
Consistent Systems
Multiple Consistent Systems
3.4.7 Matrix Inverses
The Inverse of a Square Nonsingular Matrix
The Inverse and the Solution to a Linear System
Inverses and Transposes
Nonsquare Full Rank Matrices: Right and Left Inverses
3.4.8 Full Rank Factorization
3.4.9 Multiplication by Full Rank Matrices
Products with a Nonsingular Matrix
Products with a General Full Rank Matrix
Preservation of Positive Definiteness
The General Linear Group
3.4.10 Nonfull Rank and Equivalent Matrices
Equivalent Canonical Forms
A Factorization Based on an Equivalent Canonical Form
Equivalent Forms of Symmetric Matrices
3.4.11 Gramian Matrices: Products of the Form ATA
General Properties of Gramian Matrices
Rank of ATA
Zero Matrices and Equations Involving Gramians
3.4.12 A Lower Bound on the Rank of a Matrix Product
3.4.13 Determinants of Inverses
3.4.14 Inverses of Products and Sums of Nonsingular Matrices
Inverses of Cayley Products of Matrices
Inverses of Kronecker Products of Matrices
Inverses of Sums of Matrices and Their Inverses
An Expansion of a Matrix Inverse
3.4.15 Inverses of Matrices with Special Forms
3.4.16 Determining the Rank of a Matrix
3.5 The Schur Complement in Partitioned Square Matrices
3.5.1 Inverses of Partitioned Matrices
3.5.2 Determinants of Partitioned Matrices
3.6 Linear Systems of Equations
3.6.1 Solutions of Linear Systems
Underdetermined Systems
Overdetermined Systems
Solutions in Consistent Systems: Generalized Inverses
3.6.2 Null Space: The Orthogonal Complement
3.6.3 Orthonormal Completion
3.7 Generalized Inverses
3.7.1 Immediate Properties of Generalized Inverses
Rank and the Reflexive Generalized Inverse
Properties of Generalized Inverses Useful in Analysis of Linear Models
3.7.2 The Moore–Penrose Inverse
Definitions and Terminology
Existence
Uniqueness
Other Properties
3.7.3 Generalized Inverses of Products and Sums of Matrices
3.7.4 Generalized Inverses of Partitioned Matrices
3.8 Orthogonality
3.8.1 Orthogonal Matrices: Definition and Simple Properties
3.8.2 Unitary Matrices
3.8.3 Orthogonal and Orthonormal Columns
3.8.4 The Orthogonal Group
3.8.5 Conjugacy
3.9 Eigenanalysis: Canonical Factorizations
3.9.1 Eigenvalues and Eigenvectors Are Remarkable
3.9.2 Basic Properties of Eigenvalues and Eigenvectors
Eigenvalues of Elementary Operator Matrices
Left Eigenvectors
3.9.3 The Characteristic Polynomial
How Many Eigenvalues Does a Matrix Have?
Properties of the Characteristic Polynomial
Additional Properties of Eigenvalues and Eigenvectors
Eigenvalues and the Trace and the Determinant
3.9.4 The Spectrum
Notation
The Spectral Radius
Linear Independence of Eigenvectors Associated with Distinct Eigenvalues
The Eigenspace and Geometric Multiplicity
Algebraic Multiplicity
Gershgorin Disks
3.9.5 Similarity Transformations
Orthogonally and Unitarily Similar Transformations
Uses of Similarity Transformations
3.9.6 Schur Factorization
3.9.7 Similar Canonical Factorization: Diagonalizable Matrices
Symmetric Matrices
A Defective Matrix
The Jordan Decomposition
3.9.8 Properties of Diagonalizable Matrices
Matrix Functions
3.9.9 Eigenanalysis of Symmetric Matrices
Orthogonality of Eigenvectors: Orthogonal Diagonalization
Spectral Decomposition
Kronecker Products of Symmetric Matrices: Orthogonal Diagonalization
Quadratic Forms and the Rayleigh Quotient
The Fourier Expansion
Powers of a Symmetric Matrix
The Trace and the Sum of the Eigenvalues
3.9.10 Generalized Eigenvalues and Eigenvectors
Matrix Pencils
3.9.11 Singular Values and the Singular Value Decomposition (SVD)
The Fourier Expansion in Terms of the Singular Value Decomposition
The Singular Value Decomposition and the Orthogonally Diagonal Factorization
3.10 Positive Definite and Nonnegative Definite Matrices
3.10.1 Eigenvalues of Positive and Nonnegative Definite Matrices
3.10.2 Inverse of Positive Definite Matrices
3.10.3 Diagonalization of Positive Definite Matrices
3.10.4 Square Roots of Positive and Nonnegative Definite Matrices
3.11 Matrix Norms
3.11.1 Matrix Norms Induced from Vector Norms
Lp Matrix Norms
L1, L2, and L∞ Norms of Symmetric Matrices
3.11.2 The Frobenius Norm—The ``Usual'' Norm
The Frobenius Norm and the Singular Values
3.11.3 Other Matrix Norms
3.11.4 Matrix Norm Inequalities
3.11.5 The Spectral Radius
3.11.6 Convergence of a Matrix Power Series
Conditions for Convergence of a Sequence of Powers to 0
Another Perspective on the Spectral Radius: Relation to Norms
Convergence of a Power Series: Inverse of I-A
Nilpotent Matrices
3.12 Approximation of Matrices
3.12.1 Measures of the Difference between Two Matrices
3.12.2 Best Approximation with a Matrix of Given Rank
Appendix: Matrices in R
Exercises
4 Matrix Transformations and Factorizations
Factorizations
Computational Methods: Direct and Iterative
4.1 Linear Geometric Transformations
4.1.1 Invariance Properties of Linear Transformations
4.1.2 Transformations by Orthogonal Matrices
4.1.3 Rotations
4.1.4 Reflections
4.1.5 Translations; Homogeneous Coordinates
4.2 Householder Transformations (Reflections)
4.2.1 Zeroing All Elements but One in a Vector
4.2.2 Computational Considerations
4.3 Givens Transformations (Rotations)
4.3.1 Zeroing One Element in a Vector
4.3.2 Givens Rotations That Preserve Symmetry
4.3.3 Givens Rotations to Transform to Other Values
4.3.4 Fast Givens Rotations
4.4 Factorization of Matrices
4.5 LU and LDU Factorizations
4.5.1 Properties: Existence
4.5.2 Pivoting
4.5.3 Use of Inner Products
4.5.4 Properties: Uniqueness
4.5.5 Properties of the LDU Factorization of a Square Matrix
4.6 QR Factorization
4.6.1 Related Matrix Factorizations
4.6.2 Matrices of Full Column Rank
Relation to the Moore-Penrose Inverse for Matrices of Full Column Rank
4.6.3 Nonfull Rank Matrices
Relation to the Moore-Penrose Inverse
4.6.4 Determining the Rank of a Matrix
4.6.5 Formation of the QR Factorization
4.6.6 Householder Reflections to Form the QR Factorization
4.6.7 Givens Rotations to Form the QR Factorization
4.6.8 Gram-Schmidt Transformations to Form the QR Factorization
4.7 Factorizations of Nonnegative Definite Matrices
4.7.1 Square Roots
4.7.2 Cholesky Factorization
Cholesky Decomposition of Singular Nonnegative Definite Matrices
Relations to Other Factorizations
4.7.3 Factorizations of a Gramian Matrix
4.8 Approximate Matrix Factorization
4.8.1 Nonnegative Matrix Factorization
4.8.2 Incomplete Factorizations
Appendix: R Functions for Matrix Computations and for Graphics
Exercises
5 Solution of Linear Systems
5.1 Condition of Matrices
5.1.1 Condition Number
5.1.2 Improving the Condition Number
Ridge Regression and the Condition Number
5.1.3 Numerical Accuracy
5.2 Direct Methods for Consistent Systems
5.2.1 Gaussian Elimination and Matrix Factorizations
Pivoting
Nonfull Rank and Nonsquare Systems
5.2.2 Choice of Direct Method
5.3 Iterative Methods for Consistent Systems
5.3.1 The Gauss-Seidel Method with Successive Overrelaxation
Convergence of the Gauss-Seidel Method
Successive Overrelaxation
5.3.2 Conjugate Gradient Methods for Symmetric Positive Definite Systems
The Conjugate Gradient Method
Krylov Methods
GMRES Methods
Preconditioning
5.3.3 Multigrid Methods
5.4 Iterative Refinement
5.5 Updating a Solution to a Consistent System
5.6 Overdetermined Systems: Least Squares
5.6.1 Least Squares Solution of an Overdetermined System
Orthogonality of Least Squares Residuals to span(X)
Numerical Accuracy in Overdetermined Systems
5.6.2 Least Squares with a Full Rank Coefficient Matrix
5.6.3 Least Squares with a Coefficient Matrix Not of Full Rank
An Optimal Property of the Solution Using the Moore-Penrose Inverse
5.6.4 Weighted Least Squares
5.6.5 Updating a Least Squares Solution of an Overdetermined System
5.7 Other Solutions of Overdetermined Systems
5.7.1 Solutions That Minimize Other Norms of the Residuals
Minimum L1 Norm Fitting: Least Absolute Values
Minimum L∞ Norm Fitting: Minimax
Lp Norms and Iteratively Reweighted Least Squares
5.7.2 Regularized Solutions
5.7.3 Other Restrictions on the Solutions
5.7.4 Minimizing Orthogonal Distances
Appendix: R Functions for Solving Linear Systems
Exercises
6 Evaluation of Eigenvalues and Eigenvectors
6.1 General Computational Methods
6.1.1 Numerical Condition of an Eigenvalue Problem
6.1.2 Eigenvalues from Eigenvectors and Vice Versa
6.1.3 Deflation
Deflation of Symmetric Matrices
6.1.4 Preconditioning
6.1.5 Shifting
6.2 Power Method
6.3 Jacobi Method
6.4 QR Method
6.5 Krylov Methods
6.6 Generalized Eigenvalues
6.7 Singular Value Decomposition
Exercises
7 Real Analysis and Probability Distributions of Vectors and Matrices
7.1 Basics of Differentiation
7.1.1 Continuity
7.1.2 Notation and Properties
7.1.3 Differentials
7.1.4 Use of Differentiation in Optimization
7.2 Types of Differentiation
7.2.1 Differentiation with Respect to a Scalar
Derivatives of Vectors with Respect to Scalars
Derivatives of Matrices with Respect to Scalars
Derivatives of Functions with Respect to Scalars
Higher-Order Derivatives with Respect to Scalars
7.2.2 Differentiation with Respect to a Vector
Derivatives of Scalars with Respect to Vectors; The Gradient
Derivatives of Vectors with Respect to Vectors; The Jacobian
Derivatives of Vectors with Respect to Vectors in IR3; The Divergence and the Curl
Derivatives of Matrices with Respect to Vectors
Higher-Order Derivatives with Respect to Vectors; The Hessian
Summary of Derivatives with Respect to Vectors
7.2.3 Differentiation with Respect to a Matrix
7.3 Integration
7.3.1 Multidimensional Integrals and Integrals Involving Vectors and Matrices
7.3.2 Change of Variables; The Jacobian
7.3.3 Integration Combined with Other Operations
7.4 Multivariate Probability Theory
7.4.1 Random Variables and Probability Distributions
The Distribution Function and Probability Density Function
Expected Values; The Expectation Operator
Expected Values; Generating Functions
Vector Random Variables
Matrix Random Variables
Special Random Variables
7.4.2 Distributions of Transformations of Random Variables
Change-of-Variables Method
Inverse CDF Method
Moment-Generating Function Method
7.4.3 The Multivariate Normal Distribution
Linear Transformations of a Multivariate Normal Random Variable
The Matrix Normal Distribution
7.4.4 Distributions Derived from the Multivariate Normal
7.4.5 Chi-Squared Distributions
The Family of Distributions Nn(0,σ2In)
The Family of Distributions Nd(μ,Σ)
7.4.6 Wishart Distributions
7.5 Multivariate Random Number Generation
7.5.1 The Multivariate Normal Distribution
7.5.2 Random Correlation Matrices
Appendix: R for Working with Probability Distributions and for Simulating Random Data
Exercises
Part II Applications in Statistics and Data Science
8 Special Matrices and Operations Useful in Modeling and Data Science
8.1 Data Matrices and Association Matrices
8.1.1 Flat Files
8.1.2 Graphs and Other Data Structures
Adjacency Matrix: Connectivity Matrix
Digraphs
Connectivity of Digraphs
Irreducible Matrices
Strong Connectivity of Digraphs and Irreducibility of Matrices
8.1.3 Term-by-Document Matrices
8.1.4 Sparse Matrices
8.1.5 Probability Distribution Models
8.1.6 Derived Association Matrices
8.2 Symmetric Matrices and Other Unitarily Diagonalizable Matrices
8.2.1 Some Important Properties of Symmetric Matrices
8.2.2 Approximation of Symmetric Matrices and an Important Inequality
8.2.3 Normal Matrices
8.3 Nonnegative Definite Matrices: Cholesky Factorization
8.3.1 Eigenvalues of Nonnegative Definite Matrices
8.3.2 The Square Root and the Cholesky Factorization
8.3.3 The Convex Cone of Nonnegative Definite Matrices
8.4 Positive Definite Matrices
8.4.1 Leading Principal Submatrices of Positive Definite Matrices
8.4.2 The Convex Cone of Positive Definite Matrices
8.4.3 Inequalities Involving Positive Definite Matrices
8.5 Idempotent and Projection Matrices
8.5.1 Idempotent Matrices
Symmetric Idempotent Matrices
Cochran's Theorem
8.5.2 Projection Matrices: Symmetric Idempotent Matrices
Projections onto Linear Combinations of Vectors
8.6 Special Matrices Occurring in Data Analysis
8.6.1 Gramian Matrices
Sums of Squares and Cross-Products
Some Immediate Properties of Gramian Matrices
Generalized Inverses of Gramian Matrices
Eigenvalues of Gramian Matrices
8.6.2 Projection and Smoothing Matrices
A Projection Matrix Formed from a Gramian Matrix
Smoothing Matrices
Effective Degrees of Freedom
Residuals from Smoothed Data
8.6.3 Centered Matrices and Variance-Covariance Matrices
Centering and Scaling of Data Matrices
Gramian Matrices Formed from Centered Matrices: Covariance Matrices
Gramian Matrices Formed from Scaled Centered Matrices: Correlation Matrices
8.6.4 The Generalized Variance
Comparing Variance-Covariance Matrices
8.6.5 Similarity Matrices
8.6.6 Dissimilarity Matrices
8.7 Nonnegative and Positive Matrices
The Convex Cones of Nonnegative and Positive Matrices
8.7.1 Properties of Square Positive Matrices
8.7.2 Irreducible Square Nonnegative Matrices
Properties of Square Irreducible Nonnegative Matrices; the Perron-Frobenius Theorem
Primitive Matrices
Limiting Behavior of Primitive Matrices
8.7.3 Stochastic Matrices
8.7.4 Leslie Matrices
8.8 Other Matrices with Special Structures
8.8.1 Helmert Matrices
8.8.2 Vandermonde Matrices
8.8.3 Hadamard Matrices and Orthogonal Arrays
8.8.4 Toeplitz Matrices
Inverses of Certain Toeplitz Matrices and Other Banded Matrices
8.8.5 Circulant Matrices
8.8.6 Fourier Matrices and the Discrete Fourier Transform
Fourier Matrices and Elementary Circulant Matrices
The Discrete Fourier Transform
8.8.7 Hankel Matrices
8.8.8 Cauchy Matrices
8.8.9 Matrices Useful in Graph Theory
Adjacency Matrix: Connectivity Matrix
Digraphs
Use of the Connectivity Matrix
The Laplacian Matrix of a Graph
8.8.10 Z-Matrices and M-Matrices
Exercises
9 Selected Applications in Statistics
Structure in Data and Statistical Data Analysis
9.1 Linear Models
Notation
Statistical Inference
9.1.1 Fitting the Model
Ordinary Least Squares
Weighted Least Squares
Variations on the Criteria for Fitting
Regularized Fits
Orthogonal Distances
Collinearity
9.1.2 Least Squares Fit of Full-Rank Models
9.1.3 Least Squares Fits of Nonfull-Rank Models
A Classification Model: Numerical Example
Fitting the Model Using Generalized Inverses
Uniqueness
9.1.4 Computing the Solution
Direct Computations on X
The Normal Equations and the Sweep Operator
Computations for Analysis of Variance
9.1.5 Properties of a Least Squares Fit
Geometrical Properties
Degrees of Freedom
The Hat Matrix and Leverage
9.1.6 Linear Least Squares Subject to Linear Equality Constraints
9.1.7 Weighted Least Squares
9.1.8 Updating Linear Regression Statistics
Adding More Variables
Adding More Observations
Adding More Observations Using Weights
9.1.9 Linear Smoothing
9.1.10 Multivariate Linear Models
Fitting the Model
Partitioning the Sum of Squares
9.2 Statistical Inference in Linear Models
Statistical Properties of Estimators
Full-Rank and Nonfull-Rank Models
9.2.1 The Probability Distribution of ε
Expectation of ε
Variances of ε and of the Least Squares Fits
Normality: εNn(0,σ2In)
Maximum Likelihood Estimators
9.2.2 Estimability
Uniqueness and Unbiasedness of Least Squares Estimators
Variance of Least Squares Estimators of Estimable Combinations
The Classification Model Numerical Example (Continued from Page ??
9.2.3 The Gauss-Markov Theorem
9.2.4 Testing Linear Hypotheses
9.2.5 Statistical Inference in Linear Models with Heteroscedastic or Correlated Errors
9.2.6 Statistical Inference for Multivariate Linear Models
9.3 Principal Components
9.3.1 Principal Components of a Random Vector
9.3.2 Principal Components of Data
Principal Components Directly from the Data Matrix
Dimension Reduction
9.4 Condition of Models and Data
9.4.1 Ill-Conditioning in Statistical Applications
9.4.2 Variable Selection
9.4.3 Principal Components Regression
9.4.4 Shrinkage Estimation
Ridge Regression
Lasso Regression
Elastic Net
9.4.5 Statistical Inference About the Rank of a Matrix
Numerical Approximation and Statistical Inference
Statistical Tests of the Rank of a Class of Matrices
Statistical Tests of the Rank Based on an LDU Factorization
9.4.6 Incomplete Data
9.5 Stochastic Processes
9.5.1 Markov Chains
Properties of Markov Chains
Limiting Behavior of Markov Chains
9.5.2 Markovian Population Models
9.5.3 Autoregressive Processes
Relation of the Autocorrelations to the Autoregressive Coefficients
9.6 Optimization of Scalar-Valued Functions
9.6.1 Stationary Points of Functions
9.6.2 Newton's Method
Quasi-Newton Methods
9.6.3 Least Squares
Linear Least Squares
Nonlinear Least Squares: The Gauss-Newton Method
Levenberg-Marquardt Method
9.6.4 Maximum Likelihood
9.6.5 Optimization of Functions with Constraints
Equality-Constrained Linear Least Squares Problems
The Reduced Gradient and Reduced Hessian
Lagrange Multipliers
The Lagrangian
Another Example: The Rayleigh Quotient
Optimization of Functions with Inequality Constraints
Inequality-Constrained Linear Least Squares Problems
Nonlinear Least Squares as an Inequality-Constrained Problem
9.6.6 Optimization Without Differentiation
Appendix: R for Applications in Statistics
Exercises
Part III Numerical Methods and Software
10 Numerical Methods
10.1 Software Development
10.1.1 Standards
10.1.2 Coding Systems
10.1.3 Types of Data
10.1.4 Missing Data
10.1.5 Data Structures
10.1.6 Computer Architectures and File Systems
10.2 Digital Representation of Numeric Data
10.2.1 The Fixed-Point Number System
Software Representation and Big Integers
10.2.2 The Floating-Point Model for Real Numbers
The Parameters of the Floating-Point Representation
Standardization of Floating-Point Representation
Special Floating-Point Numbers
10.2.3 Language Constructs for Representing Numeric Data
C
Fortran
Determining the Numerical Characteristics of a Particular Computer
10.2.4 Other Variations in the Representation of Data: Portability of Data
10.3 Computer Operations on Numeric Data
10.3.1 Fixed-Point Operations
10.3.2 Floating-Point Operations
Errors
Guard Digits and Chained Operations
Addition of Several Numbers
Compensated Summation
Catastrophic Cancellation
Standards for Floating-Point Operations
Operations Involving Special Floating-Point Numbers
Comparison of Real Numbers and Floating-Point Numbers
10.3.3 Language Constructs for Operations on Numeric Data
10.3.4 Software Methods for Extending the Precision
Multiple Precision
Rational Fractions
Interval Arithmetic
10.3.5 Exact Computations
Exact Dot Product (EDP)
10.4 Numerical Algorithms and Analysis
Algorithms and Programs
10.4.1 Error in Numerical Computations
Measures of Error and Bounds for Errors
Error of Approximation
Algorithms and Data
Condition of Data
Robustness of Algorithms
Stability of Algorithms
Reducing the Error in Numerical Computations
10.4.2 Efficiency
Measuring Efficiency: Counting Computations
Measuring Efficiency: Timing Computations
Improving Efficiency
Scalability
Bottlenecks and Limits
High-Performance Computing
Computations in Parallel
10.4.3 Iterations and Convergence
Extrapolation
10.4.4 Computations Without Storing Data
10.4.5 Other Computational Techniques
Recursion
MapReduce
Appendix: Numerical Computations in R
Exercises
11 Numerical Linear Algebra
11.1 Computer Storage of Vectors and Matrices
11.1.1 Storage Modes
11.1.2 Strides
11.1.3 Sparsity
11.2 General Computational Considerations for Vectors and Matrices
11.2.1 Relative Magnitudes of Operands
Condition
Pivoting
``Modified'' and ``Classical'' Gram-Schmidt Transformations
11.2.2 Iterative Methods
Preconditioning
Restarting and Rescaling
Preservation of Sparsity
Iterative Refinement
11.2.3 Assessing Computational Errors
Assessing Errors in Given Computations
11.3 Multiplication of Vectors and Matrices
11.3.1 Strassen's Algorithm
11.3.2 Matrix Multiplication Using MapReduce
11.4 Other Matrix Computations
11.4.1 Rank Determination
11.4.2 Computing the Determinant
11.4.3 Computing the Condition Number
Exercises
12 Software for Numerical Linear Algebra
12.1 General Considerations
12.1.1 Software Development and Open-Source Software
12.1.2 Integrated Development, Collaborative Research, and Version Control
12.1.3 Finding Software
12.1.4 Software Design
Interoperability
Efficiency
Writing Mathematics and Writing Programs
Numerical Mathematical Objects and Computer Objects
Other Mathematical Objects and Computer Objects
Software for Statistical Applications
Robustness
Computing Paradigms: Parallel Processing
Array Structures and Indexes
Matrix Storage Modes
Storage Schemes for Sparse Matrices
12.1.5 Software Development, Maintenance, and Testing
Test Data
Assessing the Accuracy of a Computed Result
Software Reviews
12.1.6 Reproducible Research
12.2 Software Libraries
12.2.1 BLAS
12.2.2 Level 2 and Level 3 BLAS, LAPACK, and Related Libraries
12.2.3 Libraries for High-Performance Computing
Libraries for Parallel Processing
Parallel Computations in R
Graphical Processing Units
Clusters of Computers and Cloud Computing
12.2.4 The IMSL Libraries
Examples of Use of the IMSL Libraries
12.3 General-Purpose Languages and Programming Systems
12.3.1 Programming Considerations
Reverse Communication in Iterative Algorithms
Computational Efficiency
12.3.2 Modern Fortran
12.3.3 C and C++
12.3.4 Python
12.3.5 MATLAB and Octave
Appendix: R Software for Numerical Linear Algebra
Exercises
Appendices
A Notation and Definitions
A.1 General Notation
A.2 Computer Number Systems
A.3 General Mathematical Functions and Operators
A.4 Linear Spaces and Matrices
A.4.1 Norms and Inner Products
A.4.2 Matrix Shaping Notation
A.4.3 Notation for Rows or Columns of Matrices
A.4.4 Notation Relating to Matrix Determinants
A.4.5 Matrix-Vector Differentiation
A.4.6 Special Vectors and Matrices
A.4.7 Elementary Operator Matrices
A.5 Models and Data
Bibliography
Index