Inside Deep Learning: Math, Algorithms, Models

brief contents
contents
foreword
preface
acknowledgments
about this book
Who should read this book?
How this book is organized: A road map
About the mathematical notations
About the exercises
About Google Colab
About the code
liveBook discussion forum
Other online resources
about the author
about the cover illustration
Part 1 Foundational methods
1 The mechanics of learning
1.1 Getting started with Colab
1.2 The world as tensors
1.2.1 PyTorch GPU acceleration
1.3 Automatic differentiation
1.3.1 Using derivatives to minimize losses
1.3.2 Calculating a derivative with automatic differentiation
1.3.3 Putting it together: Minimizing a function with derivatives
1.4 Optimizing parameters
1.5 Loading dataset objects
1.5.1 Creating a training and testing split
Exercises
Summary
2 Fully connected networks
2.1 Neural networks as optimization
2.1.1 Notation of training a neural network
2.1.2 Building a linear regression model
2.1.3 The training loop
2.1.4 Defining a dataset
2.1.5 Defining the model
2.1.6 Defining the loss function
2.1.7 Putting it together: Training a linear regression model on the data
2.2 Building our first neural network
2.2.1 Notation for a fully connected network
2.2.2 A fully connected network in PyTorch
2.2.3 Adding nonlinearities
2.3 Classification problems
2.3.1 Classification toy problem
2.3.2 Classification loss function
2.3.3 Training a classification network
2.4 Better training code
2.4.1 Custom metrics
2.4.2 Training and testing passes
2.4.3 Saving checkpoints
2.4.4 Putting it all together: A better model training function
2.5 Training in batches
Exercises
Summary
3 Convolutional neural networks
3.1 Spatial structural prior beliefs
3.1.1 Loading MNIST with PyTorch
3.2 What are convolutions?
3.2.1 1D convolutions
3.2.2 2D convolutions
3.2.3 Padding
3.2.4 Weight sharing
3.3 How convolutions benefit image processing
3.4 Putting it into practice: Our first CNN
3.4.1 Making a convolutional layer with multiple filters
3.4.2 Using multiple filters per layer
3.4.3 Mixing convolutional layers with linear layers via flattening
3.4.4 PyTorch code for our first CNN
3.5 Adding pooling to mitigate object movement
3.5.1 CNNs with max pooling
3.6 Data augmentation
Exercises
Summary
4 Recurrent neural networks
4.1 Recurrent neural networks as weight sharing
4.1.1 Weight sharing for a fully connected network
4.1.2 Weight sharing over time
4.2 RNNs in PyTorch
4.2.1 A simple sequence classification problem
4.2.2 Embedding layers
4.2.3 Making predictions using the last time step
4.3 Improving training time with packing
4.3.1 Pad and pack
4.3.2 Packable embedding layer
4.3.3 Training a batched RNN
4.3.4 Simultaneous packed and unpacked inputs
4.4 More complex RNNs
4.4.1 Multiple layers
4.4.2 Bidirectional RNNs
Exercises
Summary
5 Modern training techniques
5.1 Gradient descent in two parts
5.1.1 Adding a learning rate schedule
5.1.2 Adding an optimizer
5.1.3 Implementing optimizers and schedulers
5.2 Learning rate schedules
5.2.1 Exponential decay: Smoothing erratic training
5.2.2 Step drop adjustment: Better smoothing
5.2.3 Cosine annealing: Greater accuracy but less stability
5.2.4 Validation plateau: Data-based adjustments
5.2.5 Comparing the schedules
5.3 Making better use of gradients
5.3.1 SGD with momentum: Adapting to gradient consistency
5.3.2 Adam: Adding variance to momentum
5.3.3 Gradient clipping: Avoiding exploding gradients
5.4 Hyperparameter optimization with Optuna
5.4.1 Optuna
5.4.2 Optuna with PyTorch
5.4.3 Pruning trials with Optuna
Exercises
Summary
6 Common design building blocks
6.1 Better activation functions
6.1.1 Vanishing gradients
6.1.2 Rectified linear units (ReLUs): Avoiding vanishing gradients
6.1.3 Training with LeakyReLU activations
6.2 Normalization layers: Magically better convergence
6.2.1 Where do normalization layers go?
6.2.2 Batch normalization
6.2.3 Training with batch normalization
6.2.4 Layer normalization
6.2.5 Training with layer normalization
6.2.6 Which normalization layer to use?
6.2.7 A peculiarity of layer normalization
6.3 Skip connections: A network design pattern
6.3.1 Implementing fully connected skips
6.3.2 Implementing convolutional skips
6.4 1 1 Convolutions: Sharing and reshaping information in channels
6.4.1 Training with 1 1 convolutions
6.5 Residual connections
6.5.1 Residual blocks
6.5.2 Implementing residual blocks
6.5.3 Residual bottlenecks
6.5.4 Implementing residual bottlenecks
6.6 Long short-term memory RNNs
6.6.1 RNNs: A fast review
6.6.2 LSTMs and the gating mechanism
6.6.3 Training an LSTM
Exercises
Summary
Part 2 Building advanced networks
7 Autoencoding and self-supervision
7.1 How autoencoding works
7.1.1 Principle component analysis is a bottleneck autoencoder
7.1.2 Implementing PCA
7.1.3 Implementing PCA with PyTorch
7.1.4 Visualizing PCA results
7.1.5 A simple nonlinear PCA
7.2 Designing autoencoding neural networks
7.2.1 Implementing an autoencoder
7.2.2 Visualizing autoencoder results
7.3 Bigger autoencoders
7.3.1 Robustness to noise
7.4 Denoising autoencoders
7.4.1 Denoising with Gaussian noise
7.5 Autoregressive models for time series and sequences
7.5.1 Implementing the char-RNN autoregressive text model
7.5.2 Autoregressive models are generative models
7.5.3 Changing samples with temperature
7.5.4 Faster sampling
Exercises
Summary
8 Object detection
8.1 Image segmentation
8.1.1 Nuclei detection: Loading the data
8.1.2 Representing the image segmentation problem in PyTorch
8.1.3 Building our first image segmentation network
8.2 Transposed convolutions for expanding image size
8.2.1 Implementing a network with transposed convolutions
8.3 U-Net: Looking at fine and coarse details
8.3.1 Implementing U-Net
8.4 Object detection with bounding boxes
8.4.1 Faster R-CNN
8.4.2 Using Faster R-CNN in PyTorch
8.4.3 Suppressing overlapping boxes
8.5 Using the pretrained Faster R-CNN
Exercises
Summary
9 Generative adversarial networks
9.1 Understanding generative adversarial networks
9.1.1 The loss computations
9.1.2 The GAN games
9.1.3 Implementing our first GAN
9.2 Mode collapse
9.3 Wasserstein GAN: Mitigating mode collapse
9.3.1 WGAN discriminator loss
9.3.2 WGAN generator loss
9.3.3 Implementing WGAN
9.4 Convolutional GAN
9.4.1 Designing a convolutional generator
9.4.2 Designing a convolutional discriminator
9.5 Conditional GAN
9.5.1 Implementing a conditional GAN
9.5.2 Training a conditional GAN
9.5.3 Controlling the generation with conditional GANs
9.6 Walking the latent space of GANs
9.6.1 Getting models from the Hub
9.6.2 Interpolating GAN output
9.6.3 Labeling latent dimensions
9.7 Ethics in deep learning
Exercises
Summary
10 Attention mechanisms
10.1 Attention mechanisms learn relative input importance
10.1.1 Training our baseline model
10.1.2 Attention mechanism mechanics
10.1.3 Implementing a simple attention mechanism
10.2 Adding some context
10.2.1 Dot score
10.2.2 General score
10.2.3 Additive attention
10.2.4 Computing attention weights
10.3 Putting it all together: A complete attention mechanism with context
Exercises
Summary
11 Sequence-to-sequence
11.1 Sequence-to-sequence as a kind of denoising autoencoder
11.1.1 Adding attention creates Seq2Seq
11.2 Machine translation and the data loader
11.2.1 Loading a small English-French dataset
11.3 Inputs to Seq2Seq
11.3.1 Autoregressive approach
11.3.2 Teacher-forcing approach
11.3.3 Teacher forcing vs. an autoregressive approach
11.4 Seq2Seq with attention
11.4.1 Implementing Seq2Seq
11.4.2 Training and evaluation
Exercises
Summary
12 Network design alternatives to RNNs
12.1 TorchText: Tools for text problems
12.1.1 Installing TorchText
12.1.2 Loading datasets in TorchText
12.1.3 Defining a baseline model
12.2 Averaging embeddings over time
12.2.1 Weighted average over time with attention
12.3 Pooling over time and 1D CNNs
12.4 Positional embeddings add sequence information to any model
12.4.1 Implementing a positional encoding module
12.4.2 Defining positional encoding models
12.5 Transformers: Big models for big data
12.5.1 Multiheaded attention
12.5.2 Transformer blocks
Exercises
Summary
13 Transfer learning
13.1 Transferring model parameters
13.1.1 Preparing an image dataset
13.2 Transfer learning and training with CNNs
13.2.1 Adjusting pretrained networks
13.2.2 Preprocessing for pretrained ResNet
13.2.3 Training with warm starts
13.2.4 Training with frozen weights
13.3 Learning with fewer labels
13.4 Pretraining with text
13.4.1 Transformers with the Hugging Face library
13.4.2 Freezing weights with no-grad
Exercises
Summary
14 Advanced building blocks
14.1 Problems with pooling
14.1.1 Aliasing compromises translation invariance
14.1.2 Anti-aliasing by blurring
14.1.3 Applying anti-aliased pooling
14.2 Improved residual blocks
14.2.1 Effective depth
14.2.2 Implementing ReZero
14.3 MixUp training reduces overfitting
14.3.1 Picking the mix rate
14.3.2 Implementing MixUp
Exercises
Summary
A Setting up Colab
A.1 Creating a Colab session
Adding a GPU
Testing your GPU
index
Symbols & Special Characters
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z