Practical Deep Learning for Cloud, Mobile, and Edge: Real-World AI & Computer-Vision Projects Using Python, Keras & TensorFlow

Cover
Copyright
Table of Contents
Preface
To the Backend/Frontend/Mobile Software Developer
To the Data Scientist
To the Student
To the Teacher
To the Robotics Enthusiast
What to Expect in Each Chapter
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Group Acknowledgments
Personal Acknowledgments
Chapter 1. Exploring the Landscape of Artificial Intelligence
An Apology
The Real Introduction
What Is AI?
Motivating Examples
A Brief History of AI
Exciting Beginnings
The Cold and Dark Days
A Glimmer of Hope
How Deep Learning Became a Thing
Recipe for the Perfect Deep Learning Solution
Datasets
Model Architecture
Frameworks
Hardware
Responsible AI
Bias
Accountability and Explainability
Reproducibility
Robustness
Privacy
Summary
Frequently Asked Questions
Chapter 2. What’s in the Picture: Image Classification with Keras
Introducing Keras
Predicting an Image’s Category
Investigating the Model
ImageNet Dataset
Model Zoos
Class Activation Maps
Summary
Chapter 3. Cats Versus Dogs: Transfer Learning in 30 Lines with Keras
Adapting Pretrained Models to New Tasks
A Shallow Dive into Convolutional Neural Networks
Transfer Learning
Fine Tuning
How Much to Fine Tune
Building a Custom Classifier in Keras with Transfer Learning
Organize the Data
Build the Data Pipeline
Number of Classes
Batch Size
Data Augmentation
Model Definition
Train the Model
Set Training Parameters
Start Training
Test the Model
Analyzing the Results
Further Reading
Summary
Chapter 4. Building a Reverse Image Search Engine: Understanding Embeddings
Image Similarity
Feature Extraction
Similarity Search
Visualizing Image Clusters with t-SNE
Improving the Speed of Similarity Search
Length of Feature Vectors
Reducing Feature-Length with PCA
Scaling Similarity Search with Approximate Nearest Neighbors
Approximate Nearest-Neighbor Benchmark
Which Library Should I Use?
Creating a Synthetic Dataset
Brute Force
Annoy
NGT
Faiss
Improving Accuracy with Fine Tuning
Fine Tuning Without Fully Connected Layers
Siamese Networks for One-Shot Face Verification
Case Studies
Flickr
Pinterest
Celebrity Doppelgangers
Spotify
Image Captioning
Summary
Chapter 5. From Novice to Master Predictor: Maximizing Convolutional Neural Network Accuracy
Tools of the Trade
TensorFlow Datasets
TensorBoard
What-If Tool
tf-explain
Common Techniques for Machine Learning Experimentation
Data Inspection
Breaking the Data: Train, Validation, Test
Early Stopping
Reproducible Experiments
End-to-End Deep Learning Example Pipeline
Basic Transfer Learning Pipeline
Basic Custom Network Pipeline
How Hyperparameters Affect Accuracy
Transfer Learning Versus Training from Scratch
Effect of Number of Layers Fine-Tuned in Transfer Learning
Effect of Data Size on Transfer Learning
Effect of Learning Rate
Effect of Optimizers
Effect of Batch Size
Effect of Resizing
Effect of Change in Aspect Ratio on Transfer Learning
Tools to Automate Tuning for Maximum Accuracy
Keras Tuner
AutoAugment
AutoKeras
Summary
Chapter 6. Maximizing Speed and Performance of TensorFlow: A Handy Checklist
GPU Starvation
nvidia-smi
TensorFlow Profiler + TensorBoard
How to Use This Checklist
Performance Checklist
Data Preparation
Data Reading
Data Augmentation
Training
Inference
Data Preparation
Store as TFRecords
Reduce Size of Input Data
Use TensorFlow Datasets
Data Reading
Use tf.data
Prefetch Data
Parallelize CPU Processing
Parallelize I/O and Processing
Enable Nondeterministic Ordering
Cache Data
Turn on Experimental Optimizations
Autotune Parameter Values
Data Augmentation
Use GPU for Augmentation
Training
Use Automatic Mixed Precision
Use Larger Batch Size
Use Multiples of Eight
Find the Optimal Learning Rate
Use tf.function
Overtrain, and Then Generalize
Install an Optimized Stack for the Hardware
Optimize the Number of Parallel CPU Threads
Use Better Hardware
Distribute Training
Examine Industry Benchmarks
Inference
Use an Efficient Model
Quantize the Model
Prune the Model
Use Fused Operations
Enable GPU Persistence
Summary
Chapter 7. Practical Tools, Tips, and Tricks
Installation
Training
Model
Data
Privacy
Education and Exploration
One Last Question
Chapter 8. Cloud APIs for Computer Vision: Up and Running in 15 Minutes
The Landscape of Visual Recognition APIs
Clarifai
Microsoft Cognitive Services
Google Cloud Vision
Amazon Rekognition
IBM Watson Visual Recognition
Algorithmia
Comparing Visual Recognition APIs
Service Offerings
Cost
Accuracy
Bias
Getting Up and Running with Cloud APIs
Training Our Own Custom Classifier
Top Reasons Why Our Classifier Does Not Work Satisfactorily
Comparing Custom Classification APIs
Performance Tuning for Cloud APIs
Effect of Resizing on Image Labeling APIs
Effect of Compression on Image Labeling APIs
Effect of Compression on OCR APIs
Effect of Resizing on OCR APIs
Case Studies
The New York Times
Uber
Giphy
OmniEarth
Photobucket
Staples
InDro Robotics
Summary
Chapter 9. Scalable Inference Serving on Cloud with TensorFlow Serving and KubeFlow
Landscape of Serving AI Predictions
Flask: Build Your Own Server
Making a REST API with Flask
Deploying a Keras Model to Flask
Pros of Using Flask
Cons of Using Flask
Desirable Qualities in a Production-Level Serving System
High Availability
Scalability
Low Latency
Geographic Availability
Failure Handling
Monitoring
Model Versioning
A/B Testing
Support for Multiple Machine Learning Libraries
Google Cloud ML Engine: A Managed Cloud AI Serving Stack
Pros of Using Cloud ML Engine
Cons of Using Cloud ML Engine
Building a Classification API
TensorFlow Serving
Installation
KubeFlow
Pipelines
Fairing
Installation
Price Versus Performance Considerations
Cost Analysis of Inference-as-a-Service
Cost Analysis of Building Your Own Stack
Summary
Chapter 10. AI in the Browser with TensorFlow.js and ml5.js
JavaScript-Based Machine Learning Libraries: A Brief History
ConvNetJS
Keras.js
ONNX.js
TensorFlow.js
TensorFlow.js Architecture
Running Pretrained Models Using TensorFlow.js
Model Conversion for the Browser
Training in the Browser
Feature Extraction
Data Collection
Training
GPU Utilization
ml5.js
PoseNet
pix2pix
Benchmarking and Practical Considerations
Model Size
Inference Time
Case Studies
Semi-Conductor
TensorSpace
Metacar
Airbnb’s Photo Classification
GAN Lab
Summary
Chapter 11. Real-Time Object Classification on iOS with Core ML
The Development Life Cycle for Artificial Intelligence on Mobile
A Brief History of Core ML
Alternatives to Core ML
TensorFlow Lite
ML Kit
Fritz
Apple’s Machine Learning Architecture
Domain-Based Frameworks
ML Framework
ML Performance Primitives
Building a Real-Time Object Recognition App
Conversion to Core ML
Conversion from Keras
Conversion from TensorFlow
Dynamic Model Deployment
On-Device Training
Federated Learning
Performance Analysis
Benchmarking Models on iPhones
Measuring Energy Impact
Benchmarking Load
Reducing App Size
Avoid Bundling the Model
Use Quantization
Use Create ML
Case Studies
Magic Sudoku
Seeing AI
HomeCourt
InstaSaber + YoPuppet
Summary
Chapter 12. Not Hotdog on iOS with Core ML and Create ML
Collecting Data
Approach 1: Find or Collect a Dataset
Approach 2: Fatkun Chrome Browser Extension
Approach 3: Web Scraper Using Bing Image Search API
Training Our Model
Approach 1: Use Web UI-based Tools
Approach 2: Use Create ML
Approach 3: Fine Tuning Using Keras
Model Conversion Using Core ML Tools
Building the iOS App
Further Exploration
Summary
Chapter 13. Shazam for Food: Developing Android Apps with TensorFlow Lite and ML Kit
The Life Cycle of a Food Classifier App
An Overview of TensorFlow Lite
TensorFlow Lite Architecture
Model Conversion to TensorFlow Lite
Building a Real-Time Object Recognition App
ML Kit + Firebase
Object Classification in ML Kit
Custom Models in ML Kit
Hosted Models
A/B Testing Hosted Models
Using the Experiment in Code
TensorFlow Lite on iOS
Performance Optimizations
Quantizing with TensorFlow Lite Converter
TensorFlow Model Optimization Toolkit
Fritz
A Holistic Look at the Mobile AI App Development Cycle
How Do I Collect Initial Data?
How Do I Label My Data?
How Do I Train My Model?
How Do I Convert the Model to a Mobile-Friendly Format?
How Do I Make my Model Performant?
How Do I Build a Great UX for My Users?
How Do I Make the Model Available to My Users?
How Do I Measure the Success of My Model?
How Do I Improve My Model?
How Do I Update the Model on My Users’ Phones?
The Self-Evolving Model
Case Studies
Lose It!
Portrait Mode on Pixel 3 Phones
Speaker Recognition by Alibaba
Face Contours in ML Kit
Real-Time Video Segmentation in YouTube Stories
Summary
Chapter 14. Building the Purrfect Cat Locator App with TensorFlow Object Detection API
Types of Computer-Vision Tasks
Classification
Localization
Detection
Segmentation
Approaches to Object Detection
Invoking Prebuilt Cloud-Based Object Detection APIs
Reusing a Pretrained Model
Obtaining the Model
Test Driving Our Model
Deploying to a Device
Building a Custom Detector Without Any Code
The Evolution of Object Detection
Performance Considerations
Key Terms in Object Detection
Intersection over Union
Mean Average Precision
Non-Maximum Suppression
Using the TensorFlow Object Detection API to Build Custom Models
Data Collection
Labeling the Data
Preprocessing the Data
Inspecting the Model
Training
Model Conversion
Image Segmentation
Case Studies
Smart Refrigerator
Crowd Counting
Face Detection in Seeing AI
Autonomous Cars
Summary
Chapter 15. Becoming a Maker: Exploring Embedded AI at the Edge
Exploring the Landscape of Embedded AI Devices
Raspberry Pi
Intel Movidius Neural Compute Stick
Google Coral USB Accelerator
NVIDIA Jetson Nano
FPGA + PYNQ
Arduino
A Qualitative Comparison of Embedded AI Devices
Hands-On with the Raspberry Pi
Speeding Up with the Google Coral USB Accelerator
Port to NVIDIA Jetson Nano
Comparing the Performance of Edge Devices
Case Studies
JetBot
Squatting for Metro Tickets
Cucumber Sorter
Further Exploration
Summary
Chapter 16. Simulating a Self-Driving Car Using End-to-End Deep Learning with Keras
A Brief History of Autonomous Driving
Deep Learning, Autonomous Driving, and the Data Problem
The “Hello, World!” of Autonomous Driving: Steering Through a Simulated Environment
Setup and Requirements
Data Exploration and Preparation
Identifying the Region of Interest
Data Augmentation
Dataset Imbalance and Driving Strategies
Training Our Autonomous Driving Model
Drive Data Generator
Model Definition
Deploying Our Autonomous Driving Model
Further Exploration
Expanding Our Dataset
Training on Sequential Data
Reinforcement Learning
Summary
Chapter 17. Building an Autonomous Car in Under an Hour: Reinforcement Learning with AWS DeepRacer
A Brief Introduction to Reinforcement Learning
Why Learn Reinforcement Learning with an Autonomous Car?
Practical Deep Reinforcement Learning with DeepRacer
Building Our First Reinforcement Learning
Step 1: Create Model
Step 2: Configure Training
Step 3: Model Training
Step 4: Evaluating the Performance of the Model
Reinforcement Learning in Action
How Does a Reinforcement Learning System Learn?
Reinforcement Learning Theory
Reinforcement Learning Algorithm in AWS DeepRacer
Deep Reinforcement Learning Summary with DeepRacer as an Example
Step 5: Improving Reinforcement Learning Models
Racing the AWS DeepRacer Car
Building the Track
AWS DeepRacer Single-Turn Track Template
Running the Model on AWS DeepRacer
Driving the AWS DeepRacer Vehicle Autonomously
Further Exploration
DeepRacer League
Advanced AWS DeepRacer
AI Driving Olympics
DIY Robocars
Roborace
Summary
Appendix A. A Crash Course in Convolutional Neural Networks
Machine Learning
Perceptron
Activation Functions
Neural Networks
Backpropagation
Shortcoming of Neural Networks
Desired Properties of an Image Classifier
Convolution
Pooling
Structure of a CNN
Further Exploration
Index
About the Authors
Colophon