Deep Learning Architectures - A Mathematical Approach

Foreword
Overview
Part I
Part II
Part III
Part IV
Part V
Bibliographical Remarks
Chapters Diagram
Notations and Symbols
Calculus
Linear Algebra
Probability Theory
Measure Theory
Information Theory
Differential Geometry
Neural Networks
Contents
Part I Introduction to Neural Networks
1 Introductory Problems
1.1 Water in a Sink
1.2 An Electronic Circuit
1.3 The Eight Rooks Problem
1.4 Biological Neuron
1.5 Linear Regression
1.6 The Cocktail Factory Network
1.7 An Electronic Network
1.8 Summary
1.9 Exercises
2 Activation Functions
2.1 Examples of Activation Functions
2.2 Sigmoidal Functions
2.3 Squashing Functions
2.4 Summary
2.5 Exercises
3 Cost Functions
3.1 Input, Output, and Target
3.2 The Supremum Error Function
3.3 The $L2$-Error Function
3.4 Mean Square Error Function
3.5 Cross-entropy
3.6 Kullback-Leibler Divergence
3.7 Jensen-Shannon Divergence
3.8 Maximum Mean Discrepancy
3.9 Other Cost Functions
3.10 Sample Estimation of Cost Functions
3.11 Cost Functions and Regularization
3.12 Training and Test Errors
3.13 Geometric Significance
3.14 Summary
3.15 Exercises
4 Finding Minima Algorithms
4.1 General Properties of Minima
4.1.1 Functions of a real variable
4.1.2 Functions of several real variables
4.2 Gradient Descent Algorithm
4.2.1 Level sets
4.2.2 Directional derivative
4.2.3 Method of Steepest Descent
4.2.4 Line Search Method
4.3 Kinematic Interpretation
4.4 Momentum Method
4.4.1 Kinematic Interpretation
4.4.2 Convergence conditions
4.5 AdaGrad
4.6 RMSProp
4.7 Adam
4.8 AdaMax
4.9 Simulated Annealing Method
4.9.1 Kinematic Approach for SA
4.9.2 Thermodynamic Interpretation for SA
4.10 Increasing Resolution Method
4.11 Hessian Method
4.12 Newton's Method
4.13 Stochastic Search
4.13.1 Deterministic variant
4.13.2 Stochastic variant
4.14 Neighborhood Search
4.14.1 Left and Right Search
4.14.2 Circular Search
4.14.3 Stochastic Spherical Search
4.14.4 From Local to Global
4.15 Continuous Learning
4.16 Summary
4.17 Exercises
5 Abstract Neurons
5.1 Definition and Properties
5.2 Perceptron Model
5.3 The Sigmoid Neuron
5.4 Logistic Regression
5.4.1 Default probability of a company
5.4.2 Binary Classifier
5.4.3 Learning with the square difference cost function
5.5 Linear Neuron
5.6 Adaline
5.7 Madaline
5.8 Continuum Input Neuron
5.9 Summary
5.10 Exercises
6 Neural Networks
6.1 An Example of Neural Network
6.1.1 Total variation and regularization
6.1.2 Backpropagation
6.2 General Neural Networks
6.2.1 Forward pass through the network
6.2.2 Going backwards through the network
6.2.3 Backpropagation of deltas
6.2.4 Concluding relations
6.2.5 Matrix form
6.2.6 Gradient descent algorithm
6.2.7 Vanishing gradient problem
6.2.8 Batch training
6.2.9 Definition of FNN
6.3 Weights Initialization
6.4 Strong and Weak Priors
6.5 Summary
6.6 Exercises
Part II Analytic Theory
7 Approximation Theorems
7.1 Dini's Theorem
7.2 Arzela-Ascoli's Theorem
7.3 Application to Neural Networks
7.4 Stone-Weierstrass' Theorem
7.5 Application to Neural Networks
7.6 Wiener's Tauberian Theorems
7.6.1 Learning signals in $L1(mathbbR)$
7.6.2 Learning signals in $L2(mathbbR)$
7.7 Contraction Principle
7.8 Application to Recurrent Neural Nets
7.9 Resonance Networks
7.10 Summary
7.11 Exercises
8 Learning with One-dimensional Inputs
8.1 Preliminary Results
8.2 One-Hidden Layer Perceptron Network
8.3 One-Hidden Layer Sigmoid Network
8.4 Learning with ReLU Functions
8.5 Learning with Softplus
8.6 Discrete Data
8.7 Summary
8.8 Exercises
9 Universal Approximators
9.1 Introductory Examples
9.2 General Setup
9.3 Single Hidden Layer Networks
9.3.1 Learning continuous functions $finC(In)$
9.3.2 Learning square integrable functions $finL2(In)$
9.3.3 Learning integrable functions $finL1(In)$
9.3.4 Learning measurable functions $fincalM(mathbbRn)$
9.4 Error Bounds Estimation
9.5 Learning $q$-integrable functions, $finLq(mathbbRn)$
9.6 Learning Solutions of ODEs
9.7 Summary
9.8 Exercises
10 Exact Learning
10.1 Learning Finite Support Functions
10.2 Learning with ReLU
10.2.1 Representation of maxima
10.2.2 Width versus Depth
10.3 Kolmogorov-Arnold-Sprecher Theorem
10.4 Irie and Miyake's Integral Formula
10.5 Exact Learning Not Always Possible
10.6 Continuum Number of Neurons
10.7 Approximation by Degenerate Kernels
10.8 Examples of Degenerate Kernels
10.9 Deep Networks
10.10 Summary
10.11 Exercises
Part III Information Processing
11 Information Representation
11.1 Information Content
11.2 Learnable Targets
11.3 Lost Information
11.4 Recoverable Information
11.5 Information Representation Examples
11.5.1 Information for indicator functions
11.5.2 Classical perceptron
11.5.3 Linear neuron
11.5.4 Sigmoid neuron
11.5.5 Arctangent neuron
11.5.6 Neural network of classical perceptrons
11.5.7 Triangle as a union of rectangles
11.5.8 Network of sigmoid neurons
11.5.9 More remarks
11.6 Compressible Layers
11.7 Layers Compressibility
11.8 Functional Independence
11.9 Summary
11.10 Exercises
12 Information Capacity Assessment
12.1 Entropy and Properties
12.1.1 Entropy of a random variable
12.1.2 Entropy under a change of coordinates
12.2 Entropy Flow
12.3 The Input Layer Entropy
12.4 The Linear Neuron
12.5 The Autoencoder
12.6 Conditional Entropy
12.7 The Mutual Information
12.8 Applications to Deep Neural Networks
12.8.1 Entropy Flow
12.8.2 Noisy layers
12.8.3 Independent layers
12.8.4 Compressionless layers
12.8.5 The number of features
12.8.6 Total Compression
12.9 Network Capacity
12.9.1 Types of capacity
12.9.2 The input distribution
12.9.3 The output distribution
12.9.4 The input-output tensor
12.9.5 The input-output matrix
12.9.6 The existence of network capacity
12.9.7 The Lagrange multiplier method
12.9.8 Finding the Capacity
12.9.9 Perceptron Capacity
12.10 The Information Bottleneck
12.10.1 An exact formal solution
12.10.2 The information plane
12.11 Information Processing with MNIST
12.11.1 A two-layer FNN
12.11.2 A three-layer FNN
12.11.3 The role of convolutional nets
12.12 Summary
12.13 Exercises
Part IV Geometric Theory
13 Output Manifolds
13.1 Introduction to Manifolds
13.1.1 Intrinsic and Extrinsic
13.1.2 Tangent space
13.1.3 Riemannian metric
13.1.4 Geodesics
13.1.5 Levi-Civita Connection
13.1.6 Submanifolds
13.1.7 Second Fundamental Form
13.1.8 Mean Curvature Vector Field
13.2 Relation to Neural Networks
13.3 The Parameters Space
13.4 Optimal Parameter Values
13.5 The Metric Structure
13.6 Regularization
13.6.1 Going for a smaller dimension
13.6.2 Norm regularization
13.6.3 Choosing the flattest manifold
13.6.4 Model averaging
13.6.5 Dropout
13.7 Summary
13.8 Exercises
14 Neuromanifolds
14.1 Statistical Manifolds
14.2 Fisher Information
14.3 Neuromanifold of a Neural Net
14.4 Fisher Metric for One Neuron
14.5 The Fisher Matrix and Its Inverse
14.6 The Fisher Metric Structure of a Neural Net
14.7 The Natural Gradient
14.8 The Natural Gradient Learning Algorithm
14.9 Log-likelihood and the Metric
14.10 Relation to the Kullback-Leibler Divergence
14.11 Simulated Annealing Method
14.12 Summary
14.13 Exercises
Part V Other Architectures
15 Pooling
15.1 Approximation of Continuous Functions
15.2 Translation Invariance
15.3 Information Approach
15.4 Pooling and Classification
15.5 Summary
15.6 Exercises
16 Convolutional Networks
16.1 Discrete One-dimensional Signals
16.2 Continuous One-dimensional Signals
16.3 Discrete Two-dimensional Signals
16.4 Convolutional Layers with 1-D Input
16.5 Convolutional Layers with 2-D Input
16.6 Geometry of CNNs
16.7 Equivariance and Invariance
16.7.1 Groups
16.7.2 Actions of groups on sets
16.7.3 Extension of actions to functions
16.7.4 Definition of equivariance
16.7.5 Convolution and equivariance
16.7.6 Definition of invariance
16.8 Summary
16.9 Exercises
17 Recurrent Neural Networks
17.1 States Systems
17.2 RNNs
17.3 Information in RNNs
17.4 Loss Functions
17.5 Backpropagation Through Time
17.6 The Gradient Problems
17.7 LSTM Cells
17.8 Deep RNNs
17.9 Summary
17.10 Exercises
18 Classification
18.1 Equivalence Relations
18.2 Entropy of a Partition
18.3 Decision Functions
18.4 One-hot-vector Decision Maps
18.5 Linear Separability
18.6 Convex Separability
18.7 Contraction Toward a Center
18.8 Learning Decision Maps
18.8.1 Linear Decision Maps
18.8.2 Nonlinear Decision Maps
18.9 Decision Boundaries
18.10 Summary
18.11 Exercises
19 Generative Models
19.1 The Need of Generative Models
19.2 Density Estimation
19.3 Adversarial Games
19.4 Generative Adversarial Networks
19.5 Generative Moment Matching Networks
19.6 Summary
19.7 Exercises
20 Stochastic Networks
20.1 Stochastic Neurons
20.2 Boltzmann Distribution
20.3 Boltzmann Machines
20.4 Boltzmann Learning
20.5 Computing the Boltzmann Distribution
20.6 Entropy of Boltzmann Distribution
20.7 Fisher Information
20.8 Applications of Boltzmann Machines
20.9 Summary
20.10 Exercises
Hints and Solutions
Appendix
Appendix A Set Theory
Appendix B Tensors
Appendix C Measure Theory
Appendix C.1 Information and mathfrakS-algebras
Appendix C.2 Measurable Functions
Appendix C.3 Measures
Appendix C.4 Integration in Measure
Appendix C.5 Image Measures
Appendix C.6 Indefinite Integrals
Appendix C.7 Radon-Nikodym Theorem
Appendix C.8 Egorov and Luzin's Theorems
Appendix C.9 Signed Measures
Appendix D Probability Theory
Appendix D.1 General definitions
Appendix D.2 Examples
Appendix D.3 Expectation
Appendix D.4 Variance
Appendix D.5 Information generated by random variables
Appendix D.5.1 Filtrations
Appendix D.5.2 Conditional expectations
Appendix D.6 Types of Convergence
Appendix D.6.1 Convergence in probability
Appendix D.6.2 Almost sure convergence
Appendix D.6.3 Lp-convergence
Appendix D.6.4 Weak convergence
Appendix D.7 Log-Likelihood Function
Appendix D.8 Brownian Motion
Appendix E Functional Analysis
Appendix E.1 Banach spaces
Appendix E.2 Linear Operators
Appendix E.3 Hahn-Banach Theorem
Appendix E.4 Hilbert Spaces
Appendix E.5 Representation Theorems
Appendix E.6 Fixed Point Theorem
Appendix F Real Analysis
Appendix F.1 Inverse Function Theorem
Appendix F.2 Differentiation in generalized sense
Appendix F.3 Convergence of sequences of functions
Appendix G Linear Algebra
Appendix G.1 Eigenvalues, Norm, and Inverse Matrix
Appendix G.2 Moore-Penrose Pseudoinverse
Bibliography
Index