Deep Learning: A Visual Approach

About the Author
About the Technical Reviewers
Brief Contents
Contents in Detail
Acknowledgments
Introduction
Who This Book Is For
This Book Has No Math and No Code
There Is Code, If You Want It
The Figures Are Available, Too!
Errata
About This Book
Part I: Foundational Ideas
Part II: Basic Machine Learning
Part III: Deep Learning Basics
Part IV: Deep Beyond the Basics
Final Words
Part I: Foundational Ideas
Chapter 1: An Overview of Machine Learning
Expert Systems
Supervised Learning
Unsupervised Learning
Reinforcement Learning
Deep Learning
Summary
Chapter 2: Essential Statistics
Describing Randomness
Random Variables and Probability Distributions
Some Common Distributions
Continuous Distributions
Discrete Distributions
Collections of Random Values
Expected Value
Dependence
Independent and Identically Distributed Variables
Sampling and Replacement
Selection with Replacement
Selection Without Replacement
Bootstrapping
Covariance and Correlation
Covariance
Correlation
Statistics Don’t Tell Us Everything
High-Dimensional Spaces
Summary
Chapter 3: Measuring Performance
Different Types of Probability
Dart Throwing
Simple Probability
Conditional Probability
Joint Probability
Marginal Probability
Measuring Correctness
Classifying Samples
The Confusion Matrix
Characterizing Incorrect Predictions
Measuring Correct and Incorrect
Accuracy
Precision
Recall
Precision-Recall Tradeoff
Misleading Measures
f1 Score
About These Terms
Other Measures
Constructing a Confusion Matrix Correctly
Summary
Chapter 4: Bayes’ Rule
Frequentist and Bayesian Probability
The Frequentist Approach
The Bayesian Approach
Frequentists vs. Bayesians
Frequentist Coin Flipping
Bayesian Coin Flipping
A Motivating Example
Picturing the Coin Probabilities
Expressing Coin Flips as Probabilities
Bayes’ Rule
Discussion of Bayes’ Rule
Bayes’ Rule and Confusion Matrices
Repeating Bayes’ Rule
The Posterior-Prior Loop
The Bayes Loop in Action
Multiple Hypotheses
Summary
Chapter 5: Curves and Surfaces
The Nature of Functions
The Derivative
Maximums and Minimums
Tangent Lines
Finding Minimums and Maximums with Derivatives
The Gradient
Water, Gravity, and the Gradient
Finding Maximums and Minimums with Gradients
Saddle Points
Summary
Chapter 6: Information Theory
Surprise and Context
Understanding Surprise
Unpacking Context
Measuring Information
Adaptive Codes
Speaking Morse
Customizing Morse Code
Entropy
Cross Entropy
Two Adaptive Codes
Using the Codes
Cross Entropy in Practice
Kullback–Leibler Divergence
Summary
Part II: Basic Machine Learning
Chapter 7: Classification
Two-Dimensional Binary Classification
2D Multiclass Classification
Multiclass Classification
One-Versus-Rest
One-Versus-One
Clustering
The Curse of Dimensionality
Dimensionality and Density
High-Dimensional Weirdness
Summary
Chapter 8: Training and Testing
Training
Testing the Performance
Test Data
Validation Data
Cross-Validation
k-Fold Cross-Validation
Summary
Chapter 9: Overfitting and Underfitting
Finding a Good Fit
Overfitting
Underfitting
Detecting and Addressing Overfitting
Early Stopping
Regularization
Bias and Variance
Matching the Underlying Data
High Bias, Low Variance
Low Bias, High Variance
Comparing Curves
Fitting a Line with Bayes’ Rule
Summary
Chapter 10: Data Preparation
Basic Data Cleaning
The Importance of Consistency
Types of Data
One-Hot Encoding
Normalizing and Standardizing
Normalization
Standardization
Remembering the Transformation
Types of Transformations
Slice Processing
Samplewise Processing
Featurewise Processing
Elementwise Processing
Inverse Transformations
Information Leakage in Cross-Validation
Shrinking the Dataset
Feature Selection
Dimensionality Reduction
Principal Component Analysis
PCA for Simple Images
PCA for Real Images
Summary
Chapter 11: Classifiers
Types of Classifiers
k-Nearest Neighbors
Decision Trees
Introduction to Trees
Using Decision Trees
Overfitting Trees
Splitting Nodes
Support Vector Machines
The Basic Algorithm
The SVM Kernel Trick
Naive Bayes
Comparing Classifiers
Summary
Chapter 12: Ensembles
Ensembles of Decision Trees
Bagging
Random Forests
Extra Trees
Boosting
Summary
Voting
Part III: Deep Learning Basics
Chapter 13: Neural Networks
Real Neurons
Artificial Neurons
The Perceptron
Modern Artificial Neurons
Drawing the Neurons
Feed-Forward Networks
Neural Network Graphs
Initializing the Weights
Deep Networks
Fully Connected Layers
Tensors
Preventing Network Collapse
Activation Functions
Straight-Line Functions
Step Functions
Piecewise Linear Functions
Smooth Functions
Activation Function Gallery
Comparing Activation Functions
Softmax
Summary
Chapter 14: Backpropagation
A High-Level Overview of Training
Punishing Error
A Slow Way to Learn
Gradient Descent
Getting Started
Backprop on a Tiny Neural Network
Finding Deltas for the Output Neurons
Using Deltas to Change Weights
Other Neuron Deltas
Backprop on a Larger Network
The Learning Rate
Building a Binary Classifier
Picking a Learning Rate
An Even Smaller Learning Rate
Summary
Chapter 15: Optimizers
Error as a 2D Curve
Adjusting the Learning Rate
Constant-Sized Updates
Changing the Learning Rate over Time
Decay Schedules
Updating Strategies
Batch Gradient Descent
Stochastic Gradient Descent
Mini-Batch Gradient Descent
Gradient Descent Variations
Momentum
Nesterov Momentum
Adagrad
Adadelta and RMSprop
Adam
Choosing an Optimizer
Regularization
Dropout
Batchnorm
Summary
Part IV: Beyond the Basics
Chapter 16: Convolutional Neural Networks
Introducing Convolution
Detecting Yellow
Weight Sharing
Larger Filters
Filters and Features
Padding
Multidimensional Convolution
Multiple Filters
Convolution Layers
1D Convolution
1×1 Convolutions
Changing Output Size
Pooling
Striding
Transposed Convolution
Hierarchies of Filters
Simplifying Assumptions
Finding Face Masks
Finding Eyes, Noses, and Mouths
Applying Our Filters
Summary
Chapter 17: Convnets in Practice
Categorizing Handwritten Digits
VGG16
Visualizing Filters, Part 1
Visualizing Filters, Part 2
Adversaries
Summary
Chapter 18: Autoencoders
Introduction to Encoding
Lossless and Lossy Encoding
Blending Representations
The Simplest Autoencoder
A Better Autoencoder
Exploring the Autoencoder
A Closer Look at the Latent Variables
The Parameter Space
Blending Latent Variables
Predicting from Novel Input
Convolutional Autoencoders
Blending Latent Variables
Predicting from Novel Input
Denoising
Variational Autoencoders
Distribution of Latent Variables
Variational Autoencoder Structure
Exploring the VAE
Working with the MNIST Samples
Working with Two Latent Variables
Producing New Input
Summary
Chapter 19: Recurrent Neural Networks
Working with Language
Common Natural Language Processing Tasks
Fine-Tuning and Downstream Networks
Transforming Text into Numbers
Fully Connected Prediction
Testing Our Network
Why Our Network Failed
Recurrent Neural Networks
Introducing State
Rolling Up Our Diagram
Recurrent Cells in Action
Training a Recurrent Neural Network
Long Short-Term Memory and Gated Recurrent Networks
Using Recurrent Neural Networks
Working with Sunspot Data
Generating Text
Different Architectures
Seq2Seq
Summary
Chapter 20: Attention and Transformers
Embedding
Embedding Words
ELMo
Attention
A Motivating Analogy
Self-Attention
Q/KV Attention
Multi-Head Attention
Layer Icons
Transformers
Skip Connections
Norm-Add
Positional Encoding
Assembling a Transformer
Transformers in Action
BERT and GPT-2
BERT
GPT-2
Generators Discussion
Data Poisoning
Summary
Chapter 21: Reinforcement Learning
Basic Ideas
Learning a New Game
The Structure of Reinforcement Learning
Step 1: The Agent Selects an Action
Step 2: The Environment Responds
Step 3: The Agent Updates Itself
Back to the Big Picture
Understanding Rewards
Flippers
L-Learning
The Basics
The L-Learning Algorithm
Testing Our Algorithm
Handling Unpredictability
Q-Learning
Q-Values and Updates
Q-Learning Policy
Putting It All Together
The Elephant in the Room
Q-learning in Action
SARSA
The Algorithm
SARSA in Action
Comparing Q-Learning and SARSA
The Big Picture
Summary
Chapter 22: Generative Adversarial Networks
Forging Money
Learning from Experience
Forging with Neural Networks
A Learning Round
Why Adversarial?
Implementing GANs
The Discriminator
The Generator
Training the GAN
GANs in Action
Building a Discriminator and Generator
Training Our Network
Testing Our Network
DCGANs
Challenges
Using Big Samples
Modal Collapse
Training with Generated Data
Summary
Chapter 23: Creative Applications
Deep Dreaming
Stimulating Filters
Running Deep Dreaming
Neural Style Transfer
Representing Style
Representing Content
Style and Content Together
Running Style Transfer
Generating More of This Book
Summary
Final Thoughts
References
Image Credits
Index