Deep Learning with TensorFlow 2.0 and Keras: Regression, ConvNets, GANs, RNNs, NLP & more with TF 2.0 and the Keras API

Cover
Copyright
Packt Page
Contributors
Table of Contents
Preface
Chapter 1: Neural Network Foundations with TensorFlow 2.0
What is TensorFlow (TF)?
What is Keras?
What are the most important changes in TensorFlow 2.0?
Introduction to neural networks
Perceptron
A first example of TensorFlow 2.0 code
Multi-layer perceptron – our first example of a network
Problems in training the perceptron and their solutions
Activation function – sigmoid
Activation function – tanh
Activation function – ReLU
Two additional activation functions – ELU and LeakyReLU
Activation functions
In short – what are neural networks after all?
A real example – recognizing handwritten digits
One-hot encoding (OHE)
Defining a simple neural network in TensorFlow 2.0
Running a simple TensorFlow 2.0 net and establishing a baseline
Improving the simple net in TensorFlow 2.0 with hidden layers
Further improving the simple net in TensorFlow with Dropout
Testing different optimizers in TensorFlow 2.0
Increasing the number of epochs
Controlling the optimizer learning rate
Increasing the number of internal hidden neurons
Increasing the size of batch computation
Summarizing experiments run for recognizing handwritten charts
Regularization
Adopting regularization to avoid overfitting
Understanding BatchNormalization
Playing with Google Colab – CPUs, GPUs, and TPUs
Sentiment analysis
Hyperparameter tuning and AutoML
Predicting output
A practical overview of backpropagation
What have we learned so far?
Towards a deep learning approach
References
Chapter 2: TensorFlow 1.x and 2.x
Understanding TensorFlow 1.x
TensorFlow 1.x computational graph program structure
Computational graphs
Working with constants, variables, and placeholders
Examples of operations
Constants
Sequences
Random tensors
Variables
An example of TensorFlow 1.x in TensorFlow 2.x
Understanding TensorFlow 2.x
Eager execution
AutoGraph
Keras APIs – three programming models
Sequential API
Functional API
Model subclassing
Callbacks
Saving a model and weights
Training from tf.data.datasets
tf.keras or Estimators?
Ragged tensors
Custom training
Distributed training in TensorFlow 2.x
Multiple GPUs
MultiWorkerMirroredStrategy
TPUStrategy
ParameterServerStrategy
Changes in namespaces
Converting from 1.x to 2.x
Using TensorFlow 2.x effectively
The TensorFlow 2.x ecosystem
Language bindings
Keras or tf.keras?
Summary
Chapter 3: Regression
What is regression?
Prediction using linear regression
Simple linear regression
Multiple linear regression
Multivariate linear regression
TensorFlow Estimators
Feature columns
Input functions
MNIST using TensorFlow Estimator API
Predicting house price using linear regression
Classification tasks and decision boundaries
Logistic regression
Logistic regression on the MNIST dataset
Summary
References
Chapter 4: Convolutional Neural Networks
Deep Convolutional Neural Network (DCNN)
Local receptive fields
Shared weights and bias
A mathematical example
Convnets in TensorFlow 2.x
Pooling layers
Max pooling
Average pooling
ConvNets summary
An example of DCNN ‒ LeNet
LeNet code in TensorFlow 2.0
Understanding the power of deep learning
Recognizing CIFAR-10 images with deep learning
Improving the CIFAR-10 performance with a deeper network
Improving the CIFAR-10 performance with data augmentation
Predicting with CIFAR-10
Very deep convolutional networks for large-scale image recognition
Recognizing cats with a VGG16 Net
Utilizing tf.Keras built-in VGG16 Net module
Recycling prebuilt deep learning models for extracting features
Summary
References
Chapter 5: Advanced Convolutional Neural Networks
Computer vision
Composing CNNs for complex tasks
Classification and localization
Semantic segmentation
Object detection
Instance segmentation
Classifying Fashion-MNIST with a tf.keras - estimator model
Run Fashion-MNIST the tf.keras - estimator model on GPUs
Deep Inception-v3 Net used for transfer learning
Transfer learning for classifying horses and humans
Application Zoos with tf.keras and TensorFlow Hub
Keras applications
TensorFlow Hub
Other CNN architectures
AlexNet
Residual networks
HighwayNets and DenseNets
Xception
Answering questions about images (VQA)
Style transfer
Content distance
Style distance
Creating a DeepDream network
Inspecting what a network has learned
Video
Classifying videos with pretrained nets in six different ways
Textual documents
Using a CNN for sentiment analysis
Audio and music
Dilated ConvNets, WaveNet, and NSynth
A summary of convolution operations
Basic convolutional neural networks (CNN or ConvNet)
Dilated convolution
Transposed convolution
Separable convolution
Depthwise convolution
Depthwise separable convolution
Capsule networks
So what is the problem with CNNs?
So what is new with Capsule networks?
Summary
References
Chapter 6: Generative Adversarial Networks
What is a GAN?
MNIST using GAN in TensorFlow
Deep convolutional GAN (DCGAN)
DCGAN for MNIST digits
Some interesting GAN architectures
SRGAN
CycleGAN
InfoGAN
Cool applications of GANs
CycleGAN in TensorFlow 2.0
Summary
References
Chapter 7: Word Embeddings
Word embedding ‒ origins and fundamentals
Distributed representations
Static embeddings
Word2Vec
GloVe
Creating your own embedding using gensim
Exploring the embedding space with gensim
Using word embeddings for spam detection
Getting the data
Making the data ready for use
Building the embedding matrix
Define the spam classifier
Train and evaluate the model
Running the spam detector
Neural embeddings – not just for words
Item2Vec
node2vec
Character and subword embeddings
Dynamic embeddings
Language model-based embeddings
Using BERT as a feature extractor
Fine-tuning BERT
Classifying with BERT ‒ command line
Using BERT as part of your own network
Summary
References
Chapter 8: Recurrent Neural Networks
The basic RNN cell
Backpropagation through time (BPTT)
Vanishing and exploding gradients
RNN cell variants
Long short-term memory (LSTM)
Gated recurrent unit (GRU)
Peephole LSTM
RNN variants
Bidirectional RNNs
Stateful RNNs
RNN topologies
Example ‒ One-to-Many – learning to generate text
Example ‒ Many-to-One – Sentiment Analysis
Example ‒ Many-to-Many – POS tagging
Encoder-Decoder architecture – seq2seq
Example ‒ seq2seq without attention for machine translation
Attention mechanism
Example ‒ seq2seq with attention for machine translation
Transformer architecture
Summary
References
Chapter 9: Autoencoders
Introduction to autoencoders
Vanilla autoencoders
TensorFlow Keras layers ‒ defining custom layers
Reconstructing handwritten digits using an autoencoder
Sparse autoencoder
Denoising autoencoders
Clearing images using a Denoising autoencoder
Stacked autoencoder
Convolutional autoencoder for removing noise from images
Keras autoencoder example ‒ sentence vectors
Summary
References
Chapter 10: Unsupervised Learning
Principal component analysis
PCA on the MNIST dataset
TensorFlow Embedding API
K-means clustering
K-means in TensorFlow 2.0
Variations in k-means
Self-organizing maps
Colour mapping using SOM
Restricted Boltzmann machines
Reconstructing images using RBM
Deep belief networks
Variational Autoencoders
Summary
References
Chapter 11: Reinforcement Learning
Introduction
RL lingo
Deep reinforcement learning algorithms
Reinforcement success in recent years
Introduction to OpenAI Gym
Random agent playing Breakout
Deep Q-Networks
DQN for CartPole
DQN to play a game of Atari
DQN variants
Double DQN
Dueling DQN
Rainbow
Deep deterministic policy gradient
Summary
References
Chapter 12: TensorFlow and Cloud
Deep learning on cloud
Microsoft Azure
Amazon Web Services (AWS)
Google Cloud Platform (GCP)
IBM Cloud
Virtual machines on cloud
EC2 on Amazon
Compute Instance on GCP
Virtual machine on Microsoft Azure
Jupyter Notebooks on cloud
SageMaker
Google Colaboratory
Microsoft Azure Notebooks
TensorFlow Extended for production
TFX Pipelines
TFX pipeline components
TFX libraries
TensorFlow Enterprise
Summary
References
Chapter 13: TensorFlow for Mobile and IoT and TensorFlow.js
TensorFlow Mobile
TensorFlow Lite
Quantization
FlatBuffers
Mobile converter
Mobile optimized interpreter
Supported platforms
Architecture
Using TensorFlow Lite
A generic example of application
Using GPUs and accelerators
An example of application
Pretrained models in TensorFlow Lite
Image classification
Object detection
Pose estimation
Smart reply
Segmentation
Style transfer
Text classification
Question and answering
A note about using mobile GPUs
An overview of federated learning at the edge
TensorFlow FL APIs
TensorFlow.js
Vanilla TensorFlow.js
Converting models
Pretrained models
Node.js
Summary
References
Chapter 14: An introduction to AutoML
What is AutoML?
Achieving AutoML
Automatic data preparation
Automatic feature engineering
Automatic model generation
AutoKeras
Google Cloud AutoML
Using Cloud AutoML ‒ Tables solution
Using Cloud AutoML ‒ Vision solution
Using Cloud AutoML ‒ Text Classification solution
Using Cloud AutoML ‒ Translation solution
Using Cloud AutoML ‒ Video Intelligence Classification solution
Cost
Bringing Google AutoML to Kaggle
Summary
References
Chapter 15: The Math Behind Deep Learning
History
Some mathematical tools
Derivatives and gradients everywhere
Gradient descent
Chain rule
A few differentiation rules
Matrix operations
Activation functions
Derivative of the sigmoid
Derivative of tanh
Derivative of ReLU
Backpropagation
Forward step
Backstep
Case 1 – From hidden layer to output layer
Case 2 ‒ From hidden layer to hidden layer
Limit of backpropagation
Cross entropy and its derivative
Batch gradient descent, stochastic gradient descent, and mini-batch
Batch Gradient Descent (BGD)
Stochastic Gradient Descent (SGD)
Mini-Batch Gradient Descent (MBGD)
Thinking about backpropagation and convnets
Thinking about backpropagation and RNNs
A note on TensorFlow and automatic differentiation
Summary
References
Chapter 16: Tensor Processing Unit
C/G/T processing units
CPUs and GPUs
TPUs
Three generations of TPUs and Edge TPU
First-generation TPU
Second-generation TPU
Third-generation TPU
Edge TPU
TPU performance
How to use TPUs with Colab
Checking whether TPUs are available
Loading data with tf.data
Building a model and loading it into the TPU
Using pretrained TPU models
Using TensorFlow 2.1 and nightly build
Summary
References
Other Books You May Enjoy
Index