Discover best practices, reproducible architectures, and design patterns to help guide deep learning models from the lab into production.
In Deep Learning Patterns and Practices you will learn:
• Internal functioning of modern convolutional neural networks
• Procedural reuse design pattern for CNN architectures
• Models for mobile and IoT devices
• Assembling large-scale model deployments
• Optimizing hyperparameter tuning
• Migrating a model to a production environment
The big challenge of deep learning lies in taking cutting-edge technologies from R&D labs through to production. Deep Learning Patterns and Practices is here to help. This unique guide lays out the latest deep learning insights from author Andrew Ferlitsch’s work with Google Cloud AI. In it, you'll find deep learning models presented in a unique new way: as extendable design patterns you can easily plug-and-play into your software projects. Each valuable technique is presented in a way that's easy to understand and filled with accessible diagrams and code samples.
About the technology
Discover best practices, design patterns, and reproducible architectures that will guide your deep learning projects from the lab into production. This awesome book collects and illuminates the most relevant insights from a decade of real world deep learning experience. You’ll build your skills and confidence with each interesting example.
About the book
Deep Learning Patterns and Practices is a deep dive into building successful deep learning applications. You’ll save hours of trial-and-error by applying proven patterns and practices to your own projects. Tested code samples, real-world examples, and a brilliant narrative style make even complex concepts simple and engaging. Along the way, you’ll get tips for deploying, testing, and maintaining your projects.
What's inside
• Modern convolutional neural networks
• Design pattern for CNN architectures
• Models for mobile and IoT devices
• Large-scale model deployments
• Examples for computer vision
About the reader
For machine learning engineers familiar with Python and deep learning.
About the author
Andrew Ferlitsch is an expert on computer vision, deep learning, and operationalizing ML in production at Google Cloud AI Developer Relations.
Author(s): Andrew Ferlitsch
Publisher: Manning Publications
Year: 2021
Language: English
Commentary: Vector PDF
Pages: 472
City: Shelter Island, NY
Tags: Neural Networks; Deep Learning; Python; Convolutional Neural Networks; Autoencoders; Classification; Transfer Learning; Keras; Pipelines; Hyperparameter Tuning; Best Practices; NumPy; Natural Language Understanting; HDF5; Load Balancing; ResNet; Data Pipelines; Mobile Applications; A/B Testing; TensorFlow Lite; DICOM; Backpropagation; Model Training; Data Augmentation
Deep Learning Patterns and Practices
brief contents
contents
preface
acknowledgments
about this book
Who should read this book
How this book is organized: A roadmap
About the code
liveBook discussion forum
Other online resources
about the author
about the cover illustration
Part 1: Deep learning fundamentals
Chapter 1: Designing modern machine learning
1.1 A focus on adaptability
1.1.1 Computer vision leading the way
1.1.2 Beyond computer vision: NLP, NLU, structured data
1.2 The evolution in machine learning approaches
1.2.1 Classical AI vs. narrow AI
1.2.2 Next steps in computer learning
1.3 The benefits of design patterns
Chapter 2: Deep neural networks
2.1 Neural network basics
2.1.1 Input layer
2.1.2 Deep neural networks
2.1.3 Feed-forward networks
2.1.4 Sequential API method
2.1.5 Functional API method
2.1.6 Input shape vs. input layer
2.1.7 Dense layer
2.1.8 Activation functions
2.1.9 Shorthand syntax
2.1.10 Improving accuracy with an optimizer
2.2 DNN binary classifier
2.3 DNN multiclass classifier
2.4 DNN multilabel multiclass classifier
2.5 Simple image classifier
2.5.1 Flattening
2.5.2 Overfitting and dropout
Chapter 3: Convolutional and residual neural networks
3.1 Convolutional neural networks
3.1.1 Why we use a CNN over a DNN for image models
3.1.2 Downsampling (resizing)
3.1.3 Feature detection
3.1.4 Pooling
3.1.5 Flattening
3.2 The ConvNet design for a CNN
3.3 VGG networks
3.4 ResNet networks
3.4.1 Architecture
3.4.2 Batch normalization
3.4.3 ResNet50
Chapter 4: Training fundamentals
4.1 Forward feeding and backward propagation
4.1.1 Feeding
4.1.2 Backward propagation
4.2 Dataset splitting
4.2.1 Training and test sets
4.2.2 One-hot encoding
4.3 Data normalization
4.3.1 Normalization
4.3.2 Standardization
4.4 Validation and overfitting
4.4.1 Validation
4.4.2 Loss monitoring
4.4.3 Going deeper with layers
4.5 Convergence
4.6 Checkpointing and early stopping
4.6.1 Checkpointing
4.6.2 Early stopping
4.7 Hyperparameters
4.7.1 Epochs
4.7.2 Steps
4.7.3 Batch size
4.7.4 Learning rate
4.8 Invariance
4.8.1 Translational invariance
4.8.2 Scale invariance
4.8.3 TF.Keras ImageDataGenerator
4.9 Raw (disk) datasets
4.9.1 Directory structure
4.9.2 CSV file
4.9.3 JSON file
4.9.4 Reading images
4.9.5 Resizing
4.10 Model save/restore
4.10.1 Save
4.10.2 Restore
Part 2: Basic design pattern
Chapter 5: Procedural design pattern
5.1 Basic neural network architecture
5.2 Stem component
5.2.1 VGG
5.2.2 ResNet
5.2.3 ResNeXt
5.2.4 Xception
5.3 Pre-stem
5.4 Learner component
5.4.1 ResNet
5.4.2 DenseNet
5.5 Task component
5.5.1 ResNet
5.5.2 Multilayer output
5.5.3 SqueezeNet
5.6 Beyond computer vision: NLP
5.6.1 Natural-language understanding
5.6.2 Transformer architecture
Chapter 6: Wide convolutional neural networks
Inception V1
6.1 Inception v1
6.1.1 Naive inception module
6.1.2 Inception v1 module
6.1.3 Stem
6.1.4 Learner
6.1.5 Auxiliary classifiers
6.1.6 Classifier
6.2 Inception v2: Factoring convolutions
6.3 Inception v3: Architecture redesign
6.3.1 Inception groups and blocks
6.3.2 Normal convolution
6.3.3 Spatial separable convolution
6.3.4 Stem redesign and implementation
6.3.5 Auxiliary classifier
6.4 ResNeXt: Wide residual neural networks
6.4.1 ResNeXt block
6.4.2 ResNeXt architecture
6.5 Wide residual network
6.5.1 WRN-50-2 architecture
6.5.2 Wide residual block
6.6 Beyond computer vision: Structured data
Chapter 7: Alternative connectivity patterns
7.1 DenseNet: Densely connected convolutional neural network
7.1.1 Dense group
7.1.2 Dense block
7.1.3 DenseNet macro-architecture
7.1.4 Dense transition block
7.2 Xception: Extreme Inception
7.2.1 Xception architecture
7.2.2 Entry flow of Xception
7.2.3 Middle flow of Xception
7.2.4 Exit flow of Xception
7.2.5 Depthwise separable convolution
7.2.6 Depthwise convolution
7.2.7 Pointwise convolution
7.3 SE-Net: Squeeze and excitation
7.3.1 Architecture of SE-Net
7.3.2 Group and block of SE-Net
7.3.3 SE link
Chapter 8: Mobile convolutional neural networks
8.1 MobileNet v1
8.1.1 Architecture
8.1.2 Width multiplier
8.1.3 Resolution multiplier
8.1.4 Stem
8.1.5 Learner
8.1.6 Classifier
8.2 MobileNet v2
8.2.1 Architecture
8.2.2 Stem
8.2.3 Learner
8.2.4 Classifier
8.3 SqueezeNet
8.3.1 Architecture
8.3.2 Stem
8.3.3 Learner
8.3.4 Classifier
8.3.5 Bypass connections
8.4 ShuffleNet v1
8.4.1 Architecture
8.4.2 Stem
8.4.3 Learner
8.5 Deployment
8.5.1 Quantization
8.5.2 TF Lite conversion and prediction
Chapter 9: Autoencoders
9.1 Deep neural network autoencoders
9.1.1 Autoencoder architecture
9.1.2 Encoder
9.1.3 Decoder
9.1.4 Training
9.2 Convolutional autoencoders
9.2.1 Architecture
9.2.2 Encoder
9.2.3 Decoder
9.3 Sparse autoencoders
9.4 Denoising autoencoders
9.5 Super-resolution
9.5.1 Pre-upsampling SR
9.5.2 Post-upsampling SR
9.6 Pretext tasks
9.7 Beyond computer vision: sequence to sequence
Part 3: Working with pipelines
Chapter 10: Hyperparameter tuning
10.1 Weight initialization
10.1.1 Weight distributions
10.1.2 Lottery hypothesis
10.1.3 Warm-up (numerical stability)
10.2 Hyperparameter search fundamentals
10.2.1 Manual method for hyperparameter search
10.2.2 Grid search
10.2.3 Random search
10.2.4 KerasTuner
10.3 Learning rate scheduler
10.3.1 Keras decay parameter
10.3.2 Keras learning rate scheduler
10.3.3 Ramp
10.3.4 Constant step
10.3.5 Cosine annealing
10.4 Regularization
10.4.1 Weight regularization
10.4.2 Label smoothing
10.5 Beyond computer vision
Chapter 11: Transfer learning
11.1 TF.Keras prebuilt models
11.1.1 Base model
11.1.2 Pretrained ImageNet models for prediction
11.1.3 New classifier
11.2 TF Hub prebuilt models
11.2.1 Using TF Hub pretrained models
11.2.2 New classifier
11.3 Transfer learning between domains
11.3.1 Similar tasks
11.3.2 Distinct tasks
11.3.3 Domain-specific weights
11.3.4 Domain transfer weight initialization
11.3.5 Negative transfer
11.4 Beyond computer vision
Chapter 12: Data distributions
12.1 Distribution types
12.1.1 Population distribution
12.1.2 Sampling distribution
12.1.3 Subpopulation distribution
12.2 Out of distribution
12.2.1 The MNIST curated dataset
12.2.2 Setting up the environment
12.2.3 The challenge (“in the wild”)
12.2.4 Training as a DNN
12.2.5 Training as a CNN
12.2.6 Image augmentation
12.2.7 Final test
Chapter 13: Data pipeline
13.1 Data formats and storage
13.1.1 Compressed and raw-image formats
13.1.2 HDF5 format
13.1.3 DICOM format
13.1.4 TFRecord format
13.2 Data feeding
13.2.1 NumPy
13.2.2 TFRecord
13.3 Data preprocessing
13.3.1 Preprocessing with a pre-stem
13.3.2 Preprocessing with TF Extended
13.4 Data augmentation
13.4.1 Invariance
13.4.2 Augmentation with tf.data
13.4.3 Pre-stem
Chapter 14: Training and deployment pipeline
14.1 Model feeding
14.1.1 Model feeding with tf.data.Dataset
14.1.2 Distributed feeding with tf.Strategy
14.1.3 Model feeding with TFX
14.2 Training schedulers
14.2.1 Pipeline versioning
14.2.2 Metadata
14.2.3 History
14.3 Model evaluations
14.3.1 Candidate vs. blessed model
14.3.2 TFX evaluation
14.4 Serving predictions
14.4.1 On-demand (live) serving
14.4.2 Batch prediction
14.4.3 TFX pipeline components for deployment
14.4.4 A/B testing
14.4.5 Load balancing
14.4.6 Continuous evaluation
14.5 Evolution in production pipeline design
14.5.1 Machine learning as a pipeline
14.5.2 Machine learning as a CI/CD production process
14.5.3 Model amalgamation in production
index
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Z
