Deep learning networks are getting smaller. Much smaller. The Google Assistant team can detect words with a model just 14 kilobytes in size—small enough to run on a microcontroller. With this practical book you’ll enter the field of TinyML, where deep learning and embedded systems combine to make astounding things possible with tiny devices.
Pete Warden and Daniel Situnayake explain how you can train models small enough to fit into any environment. Ideal for software and hardware developers who want to build embedded systems using machine learning, this guide walks you through creating a series of TinyML projects, step-by-step. No machine learning or microcontroller experience is necessary.
• Build a speech recognizer, a camera that detects people, and a magic wand that responds to gestures
• Work with Arduino and ultra-low-power microcontrollers
• Learn the essentials of ML and how to train your own models
• Train models to understand audio, image, and accelerometer data
• Explore TensorFlow Lite for Microcontrollers, Google’s toolkit for TinyML
• Debug applications and provide safeguards for privacy and security
• Optimize latency, energy usage, and model and binary size
Author(s): Pete Warden, Daniel Situnayake
Edition: 1
Publisher: O'Reilly Media
Year: 2019
Language: English
Commentary: True PDF
Pages: 504
City: Sebastopol, CA
Tags: Google Cloud Platform; Machine Learning; Deep Learning; C++; Debugging; Security; Python; Speech Recognition; Privacy; Keras; TensorFlow; Logging; Deployment; Application Development; Jupyter; Arduino; Microcontrollers; Gesture Recognition; Embedded Systems; STM32; Code Optimization; Object Detection; Google Colaboratory; TensorBoard; TinyML; TensorFlow Lite; Latency; Energy Usage Optimization
Cover
Copyright
Table of Contents
Preface
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Chapter 1. Introduction
Embedded Devices
Changing Landscape
Chapter 2. Getting Started
Who Is This Book Aimed At?
What Hardware Do You Need?
What Software Do You Need?
What Do We Hope You’ll Learn?
Chapter 3. Getting Up to Speed on Machine Learning
What Machine Learning Actually Is
The Deep Learning Workflow
Decide on a Goal
Collect a Dataset
Design a Model Architecture
Train the Model
Convert the Model
Run Inference
Evaluate and Troubleshoot
Wrapping Up
Chapter 4. The “Hello World” of TinyML: Building and Training a Model
What We’re Building
Our Machine Learning Toolchain
Python and Jupyter Notebooks
Google Colaboratory
TensorFlow and Keras
Building Our Model
Importing Dependencies
Generating Data
Splitting the Data
Defining a Basic Model
Training Our Model
Training Metrics
Graphing the History
Improving Our Model
Testing
Converting the Model for TensorFlow Lite
Converting to a C File
Wrapping Up
Chapter 5. The “Hello World” of TinyML: Building an Application
Walking Through the Tests
Including Dependencies
Setting Up the Test
Getting Ready to Log Data
Mapping Our Model
Creating an AllOpsResolver
Defining a Tensor Arena
Creating an Interpreter
Inspecting the Input Tensor
Running Inference on an Input
Reading the Output
Running the Tests
Project File Structure
Walking Through the Source
Starting with main_functions.cc
Handling Output with output_handler.cc
Wrapping Up main_functions.cc
Understanding main.cc
Running Our Application
Wrapping Up
Chapter 6. The “Hello World” of TinyML: Deploying to Microcontrollers
What Exactly Is a Microcontroller?
Arduino
Handling Output on Arduino
Running the Example
Making Your Own Changes
SparkFun Edge
Handling Output on SparkFun Edge
Running the Example
Testing the Program
Viewing Debug Data
Making Your Own Changes
ST Microelectronics STM32F746G Discovery Kit
Handling Output on STM32F746G
Running the Example
Making Your Own Changes
Wrapping Up
Chapter 7. Wake-Word Detection: Building an Application
What We’re Building
Application Architecture
Introducing Our Model
All the Moving Parts
Walking Through the Tests
The Basic Flow
The Audio Provider
The Feature Provider
The Command Recognizer
The Command Responder
Listening for Wake Words
Running Our Application
Deploying to Microcontrollers
Arduino
SparkFun Edge
ST Microelectronics STM32F746G Discovery Kit
Wrapping Up
Chapter 8. Wake-Word Detection: Training a Model
Training Our New Model
Training in Colab
Using the Model in Our Project
Replacing the Model
Updating the Labels
Updating command_responder.cc
Other Ways to Run the Scripts
How the Model Works
Visualizing the Inputs
How Does Feature Generation Work?
Understanding the Model Architecture
Understanding the Model Output
Training with Your Own Data
The Speech Commands Dataset
Training on Your Own Dataset
How to Record Your Own Audio
Data Augmentation
Model Architectures
Wrapping Up
Chapter 9. Person Detection: Building an Application
What We’re Building
Application Architecture
Introducing Our Model
All the Moving Parts
Walking Through the Tests
The Basic Flow
The Image Provider
The Detection Responder
Detecting People
Deploying to Microcontrollers
Arduino
SparkFun Edge
Wrapping Up
Chapter 10. Person Detection: Training a Model
Picking a Machine
Setting Up a Google Cloud Platform Instance
Training Framework Choice
Building the Dataset
Training the Model
TensorBoard
Evaluating the Model
Exporting the Model to TensorFlow Lite
Exporting to a GraphDef Protobuf File
Freezing the Weights
Quantizing and Converting to TensorFlow Lite
Converting to a C Source File
Training for Other Categories
Understanding the Architecture
Wrapping Up
Chapter 11. Magic Wand: Building an Application
What We’re Building
Application Architecture
Introducing Our Model
All the Moving Parts
Walking Through the Tests
The Basic Flow
The Accelerometer Handler
The Gesture Predictor
The Output Handler
Detecting Gestures
Deploying to Microcontrollers
Arduino
SparkFun Edge
Wrapping Up
Chapter 12. Magic Wand: Training a Model
Training a Model
Training in Colab
Other Ways to Run the Scripts
How the Model Works
Visualizing the Input
Understanding the Model Architecture
Training with Your Own Data
Capturing Data
Modifying the Training Scripts
Training
Using the New Model
Wrapping Up
Learning Machine Learning
What’s Next
Chapter 13. TensorFlow Lite for Microcontrollers
What Is TensorFlow Lite for Microcontrollers?
TensorFlow
TensorFlow Lite
TensorFlow Lite for Microcontrollers
Requirements
Why Is the Model Interpreted?
Project Generation
Build Systems
Specializing Code
Makefiles
Writing Tests
Supporting a New Hardware Platform
Printing to a Log
Implementing DebugLog()
Running All the Targets
Integrating with the Makefile Build
Supporting a New IDE or Build System
Integrating Code Changes Between Projects and Repositories
Contributing Back to Open Source
Supporting New Hardware Accelerators
Understanding the File Format
FlatBuffers
Porting TensorFlow Lite Mobile Ops to Micro
Separate the Reference Code
Create a Micro Copy of the Operator
Port the Test to the Micro Framework
Build a Bazel Test
Add Your Op to AllOpsResolver
Build a Makefile Test
Wrapping Up
Chapter 14. Designing Your Own TinyML Applications
The Design Process
Do You Need a Microcontroller, or Would a Larger Device Work?
Understanding What’s Possible
Follow in Someone Else’s Footsteps
Find Some Similar Models to Train
Look at the Data
Wizard of Oz-ing
Get It Working on the Desktop First
Chapter 15. Optimizing Latency
First Make Sure It Matters
Hardware Changes
Model Improvements
Estimating Model Latency
How to Speed Up Your Model
Quantization
Product Design
Code Optimizations
Performance Profiling
Optimizing Operations
Look for Implementations That Are Already Optimized
Write Your Own Optimized Implementation
Taking Advantage of Hardware Features
Accelerators and Coprocessors
Contributing Back to Open Source
Wrapping Up
Chapter 16. Optimizing Energy Usage
Developing Intuition
Typical Component Power Usage
Hardware Choice
Measuring Real Power Usage
Estimating Power Usage for a Model
Improving Power Usage
Duty Cycling
Cascading Design
Wrapping Up
Chapter 17. Optimizing Model and Binary Size
Understanding Your System’s Limits
Estimating Memory Usage
Flash Usage
RAM Usage
Ballpark Figures for Model Accuracy and Size on Different Problems
Speech Wake-Word Model
Accelerometer Predictive Maintenance Model
Person Presence Detection
Model Choice
Reducing the Size of Your Executable
Measuring Code Size
How Much Space Is Tensorflow Lite for Microcontrollers Taking?
OpResolver
Understanding the Size of Individual Functions
Framework Constants
Truly Tiny Models
Wrapping Up
Chapter 18. Debugging
Accuracy Loss Between Training and Deployment
Preprocessing Differences
Debugging Preprocessing
On-Device Evaluation
Numerical Differences
Are the Differences a Problem?
Establish a Metric
Compare Against a Baseline
Swap Out Implementations
Mysterious Crashes and Hangs
Desktop Debugging
Log Tracing
Shotgun Debugging
Memory Corruption
Wrapping Up
Chapter 19. Porting Models from TensorFlow to TensorFlow Lite
Understand What Ops Are Needed
Look at Existing Op Coverage in Tensorflow Lite
Move Preprocessing and Postprocessing into Application Code
Implement Required Ops if Necessary
Optimize Ops
Wrapping Up
Chapter 20. Privacy, Security, and Deployment
Privacy
The Privacy Design Document
Using a PDD
Security
Protecting Models
Deployment
Moving from a Development Board to a Product
Wrapping Up
Chapter 21. Learning More
The TinyML Foundation
SIG Micro
The TensorFlow Website
Other Frameworks
Twitter
Friends of TinyML
Wrapping Up
Appendix A. Using and Generating an Arduino Library Zip
Appendix B. Capturing Audio on Arduino
Index
About the Authors
Colophon
