A Handbook of Time-Series Analysis, Signal Processing and Dynamics

Preface......Page 25
Introduction......Page 27
The Frequency Domain and the Time Domain......Page 29
Harmonic Analysis......Page 30
Autoregressive and Moving-Average Models......Page 33
Generalised Harmonic Analysis......Page 36
The Equivalence of the Two Domains......Page 38
The Maturing of Time-Series Analysis......Page 40
Mathematical Appendix......Page 42
Polynomial Methods......Page 47
Sequences......Page 49
Linear Convolution......Page 52
Circular Convolution......Page 54
Time-Series Models......Page 56
Transfer Functions......Page 57
The Lag Operator......Page 59
Periodic Polynomials and Circular Convolution......Page 61
Polynomial Factorisation......Page 63
Complex Roots......Page 64
The Roots of Unity......Page 68
The Polynomial of Degree n......Page 69
Matrices and Polynomial Algebra......Page 71
Lower-Triangular Toeplitz Matrices......Page 72
Circulant Matrices......Page 74
The Factorisation of Circulant Matrices......Page 76
Euclid's Algorithm......Page 81
Partial Fractions......Page 85
The Expansion of a Rational Function......Page 88
Recurrence Relationships......Page 90
Laurent Series......Page 93
Analytic Functions......Page 96
Complex Line Integrals......Page 98
The Cauchy Integral Theorem......Page 100
Multiply Connected Domains......Page 102
Integrals and Derivatives of Analytic Functions......Page 103
Series Expansions......Page 104
Residues......Page 108
The Autocovariance Generating Function......Page 110
The Argument Principle......Page 112
Polynomial Computations......Page 115
Polynomials and their Derivatives......Page 116
The Division Algorithm......Page 120
Roots of Polynomials......Page 124
Real Roots......Page 125
Complex Roots......Page 130
Müller's Method......Page 135
Polynomial Interpolation......Page 140
Lagrangean Interpolation......Page 141
Divided Differences......Page 143
Difference Equations and Differential Equations......Page 147
Linear Difference Equations......Page 148
Solution of the Homogeneous Difference Equation......Page 149
Complex Roots......Page 150
Particular Solutions......Page 152
Solutions of Difference Equations with Initial Conditions......Page 155
Alternative Forms for the Difference Equation......Page 159
Linear Differential Equations......Page 161
Solution of the Homogeneous Differential Equation......Page 162
Differential Equation with Complex Roots......Page 163
Particular Solutions for Differential Equations......Page 165
Solutions of Differential Equations with Initial Conditions......Page 170
Difference and Differential Equations Compared......Page 173
Conditions for the Stability of Differential Equations......Page 174
Conditions for the Stability of Difference Equations......Page 177
The State-Space Equations......Page 187
Conversions of Difference Equations to State-Space Form......Page 189
Controllable Canonical State-Space Representations......Page 191
Observable Canonical Forms......Page 194
Reduction of State-Space Equations to a Transfer Function......Page 196
Controllability......Page 197
Observability......Page 202
Least-Squares Methods......Page 205
Matrix Computations......Page 207
Solving Linear Equations by Gaussian Elimination......Page 208
Inverting Matrices by Gaussian Elimination......Page 214
The Direct Factorisation of a Nonsingular Matrix......Page 215
The Cholesky Decomposition......Page 217
Householder Transformations......Page 221
The Q--R Decomposition of a Matrix of Full Column Rank......Page 222
The Linear Regression Model......Page 227
The Decomposition of the Sum of Squares......Page 228
Some Statistical Properties of the Estimator......Page 230
Estimating the Variance of the Disturbance......Page 231
Some Matrix Identities......Page 232
Computing a Regression via Gaussian Elimination......Page 234
Calculating the Corrected Sum of Squares......Page 237
Computing the Regression Parameters via the Q--R Decomposition......Page 241
The Normal Distribution and the Sampling Distributions......Page 244
Hypothesis Concerning the Complete Set of Coefficients......Page 245
Hypotheses Concerning a Subset of the Coefficients......Page 247
An Alternative Formulation of the F statistic......Page 249
Recursive Least-Squares Regression......Page 253
The Matrix Inversion Lemma......Page 254
Prediction Errors and Recursive Residuals......Page 255
The Updating Algorithm for Recursive Least Squares......Page 257
Initiating the Recursion......Page 261
Estimators with Limited Memories......Page 262
The Kalman Filter......Page 265
Filtering......Page 267
A Summary of the Kalman Equations......Page 270
An Alternative Derivation of the Kalman Filter......Page 271
Innovations and the Information Set......Page 273
Conditional Expectations and Dispersions of the State Vector......Page 275
The Classical Smoothing Algorithms......Page 276
Variants of the Classical Algorithms......Page 280
Multi-step Prediction......Page 283
Polynomial Regression......Page 287
The Gram--Schmidt Orthogonalisation Procedure......Page 289
A Modified Gram--Schmidt Procedure......Page 292
Uniqueness of the Gram Polynomials......Page 294
Recursive Generation of the Polynomials......Page 296
The Polynomial Regression Procedure......Page 298
Grafted Polynomials......Page 304
B-Splines......Page 307
Recursive Generation of B-spline Ordinates......Page 310
Regression with B-Splines......Page 316
Smoothing with Cubic Splines......Page 319
Cubic Spline Interpolation......Page 320
Cubic Splines and Bézier Curves......Page 327
The Minimum-Norm Property of Splines......Page 331
Smoothing Splines......Page 333
A Stochastic Model for the Smoothing Spline......Page 339
Appendix: The Wiener Process and the IMA Process......Page 345
Conditions of Optimality......Page 349
Univariate Search......Page 352
Quadratic Interpolation......Page 354
Bracketing the Minimum......Page 361
Unconstrained Optimisation via Quadratic Approximations......Page 364
The Method of Steepest Descent......Page 365
The Newton--Raphson Method......Page 366
A Modified Newton Procedure......Page 367
The Minimisation of a Sum of Squares......Page 369
Quadratic Convergence......Page 370
The Conjugate Gradient Method......Page 373
Numerical Approximations to the Gradient......Page 377
Quasi-Newton Methods......Page 378
Rank-Two Updating of the Hessian Matrix......Page 380
Fourier Methods......Page 389
Fourier Series and Fourier Integrals......Page 391
Fourier Series......Page 393
Convolution......Page 397
Fourier Approximations......Page 400
Discrete-Time Fourier Transform......Page 403
Symmetry Properties of the Fourier Transform......Page 404
The Frequency Response of a Discrete-Time System......Page 406
The Fourier Integral......Page 410
The Uncertainty Relationship......Page 412
The Delta Function......Page 414
Impulse Trains......Page 417
The Sampling Theorem......Page 418
The Frequency Response of a Continuous-Time System......Page 420
Appendix of Trigonometry......Page 422
Orthogonality Conditions......Page 423
The Discrete Fourier Transform......Page 425
Trigonometrical Representation of the DFT......Page 426
Determination of the Fourier Coefficients......Page 429
The Periodogram and Hidden Periodicities......Page 431
The Periodogram and the Empirical Autocovariances......Page 434
The Exponential Form of the Fourier Transform......Page 436
Leakage from Nonharmonic Frequencies......Page 439
The Fourier Transform and the z-Transform......Page 440
The Classes of Fourier Transforms......Page 442
Sampling in the Time Domain......Page 444
Truncation in the Time Domain......Page 447
Sampling in the Frequency Domain......Page 448
Appendix: Harmonic Cycles......Page 449
Basic Concepts......Page 453
The Two-Factor Case......Page 457
The FFT for Arbitrary Factors......Page 460
Locating the Subsequences......Page 463
The Core of the Mixed-Radix Algorithm......Page 465
Unscrambling......Page 468
The Shell of the Mixed-Radix Procedure......Page 471
The Base-2 Fast Fourier Transform......Page 473
FFT Algorithms for Real Data......Page 476
FFT for a Single Real-valued Sequence......Page 478
Time-Series Models......Page 483
Frequency Response and Transfer Functions......Page 485
Computing the Gain and Phase Functions......Page 492
The Poles and Zeros of the Filter......Page 495
Inverse Filtering and Minimum-Phase Filters......Page 501
Linear-Phase Filters......Page 503
Locations of the Zeros of Linear-Phase Filters......Page 505
FIR Filter Design by Window Methods......Page 509
Truncating the Filter......Page 513
Cosine Windows......Page 518
Design of Recursive IIR Filters......Page 522
IIR Design via Analogue Prototypes......Page 524
The Butterworth Filter......Page 525
The Chebyshev Filter......Page 527
The Bilinear Transformation......Page 530
The Butterworth and Chebyshev Digital Filters......Page 532
Frequency-Band Transformations......Page 533
Autoregressive and Moving-Average Processes......Page 539
Stationary Stochastic Processes......Page 540
Moving-Average Processes......Page 543
Computing the MA Autocovariances......Page 547
MA Processes with Common Autocovariances......Page 548
Computing the MA Parameters from the Autocovariances......Page 549
The Autocovariances and the Yule--Walker Equations......Page 554
Computing the AR Parameters......Page 561
Autoregressive Moving-Average Processes......Page 566
Calculating the ARMA Parameters from the Autocovariances......Page 571
Time-Series Analysis in the Frequency Domain......Page 575
The Filtering of White Noise......Page 576
Cyclical Processes......Page 579
The Fourier Representation of a Sequence......Page 581
The Spectral Representation of a Stationary Process......Page 582
The Autocovariances and the Spectral Density Function......Page 585
The Theorem of Herglotz and the Decomposition of Wold......Page 587
The Frequency-Domain Analysis of Filtering......Page 590
The Spectral Density Functions of ARMA Processes......Page 592
Canonical Factorisation of the Spectral Density Function......Page 596
Prediction and Signal Extraction......Page 601
Mean-Square Error......Page 602
Predicting one Series from Another......Page 603
The Technique of Prewhitening......Page 605
Extrapolation of Univariate Series......Page 606
Forecasting with ARIMA Models......Page 609
Generating the ARMA Forecasts Recursively......Page 611
Physical Analogies for the Forecast Function......Page 613
Interpolation and Signal Extraction......Page 615
Extracting the Trend from a Nonstationary Sequence......Page 617
Finite-Sample Predictions: Hilbert Space Terminology......Page 619
Recursive Prediction: The Durbin--Levinson Algorithm......Page 620
A Lattice Structure for the Prediction Errors......Page 625
Recursive Prediction: The Gram--Schmidt Algorithm......Page 627
Signal Extraction from a Finite Sample: the Stationary Case......Page 633
Signal Extraction from a Finite Sample: the Nonstationary Case......Page 635
Time-Series Estimation......Page 643
Estimating the Mean of a Stationary Process......Page 645
Asymptotic Variance of the Sample Mean......Page 647
Estimating the Autocovariances of a Stationary Process......Page 648
Asymptotic Moments of the Sample Autocovariances......Page 650
Asymptotic Moments of the Sample Autocorrelations......Page 652
Calculation of the Autocovariances......Page 655
Inefficient Estimation of the MA Autocovariances......Page 658
Efficient Estimates of the MA Autocorrelations......Page 660
Representations of the ARMA Equations......Page 663
The Least-Squares Criterion Function......Page 665
The Yule--Walker Estimates......Page 667
Estimation of MA Models......Page 668
Representations via LT Toeplitz Matrices......Page 669
Representations via Circulant Matrices......Page 671
The Gauss--Newton Estimation of the ARMA Parameters......Page 674
An Implementation of the Gauss--Newton Procedure......Page 675
Asymptotic Properties of the Least-Squares Estimates......Page 681
The Sampling Properties of the Estimators......Page 683
The Burg Estimator......Page 686
Matrix Representations of Autoregressive Models......Page 693
The AR Dispersion Matrix and its Inverse......Page 695
Density Functions of the AR Model......Page 698
The Exact M-L Estimator of an AR Model......Page 699
Conditional M-L Estimates of an AR Model......Page 702
Matrix Representations of Moving-Average Models......Page 704
The MA Dispersion Matrix and its Determinant......Page 705
Density Functions of the MA Model......Page 706
The Exact M-L Estimator of an MA Model......Page 707
Conditional M-L Estimates of an MA Model......Page 711
Matrix Representations of ARMA models......Page 712
Density Functions of the ARMA Model......Page 713
Exact M-L Estimator of an ARMA Model......Page 714
Nonparametric Estimation of the Spectral Density Function......Page 723
The Spectrum and the Periodogram......Page 724
The Expected Value of the Sample Spectrum......Page 728
Asymptotic Distribution of The Periodogram......Page 731
Smoothing the Periodogram......Page 736
Weighting the Autocovariance Function......Page 739
Weights and Kernel Functions......Page 740
Statistical Appendix: on Disc......Page 747
Multivariate Density Functions......Page 749
Functions of Random Vectors......Page 751
Expectations......Page 752
Moments of a Multivariate Distribution......Page 753
Degenerate Random Vectors......Page 755
The Multivariate Normal Distribution......Page 756
Distributions Associated with the Normal Distribution......Page 759
Quadratic Functions of Normal Vectors......Page 760
The Decomposition of a Chi-square Variate......Page 762
Limit Theorems......Page 765
Stochastic Convergence......Page 766
The Law of Large Numbers and the Central Limit Theorem......Page 771
Principles of Estimation......Page 775
Identifiability......Page 776
The Information Matrix......Page 779
The Efficiency of Estimation......Page 780
Unrestricted Maximum-Likelihood Estimation......Page 782
Restricted Maximum-Likelihood Estimation......Page 784
Tests of the Restrictions......Page 787
Index......Page 791