Simulation and synthesis are core parts of the future of AI and machine learning. Consider: programmers, data scientists, and machine learning engineers can create the brain of a self-driving car without the car. Rather than use information from the real world, you can synthesize artificial data using simulations to train traditional machine learning models. That's just the beginning.
With this practical book, you'll explore the possibilities of simulation- and synthesis-based machine learning and AI, concentrating on deep reinforcement learning and imitation learning techniques. AI and ML are increasingly data driven, and simulations are a powerful, engaging way to unlock their full potential.
You'll learn how to:
• Design an approach for solving ML and AI problems using simulations with the Unity engine
• Use a game engine to synthesize images for use as training data
• Create simulation environments designed for training deep reinforcement learning and imitation learning models
• Use and apply efficient general-purpose algorithms for simulation-based ML, such as proximal policy optimization
• Train a variety of ML models using different approaches
• Enable ML tools to work with industry-standard game development tools, using PyTorch, and the Unity ML-Agents and Perception Toolkits
Author(s): Paris Buttfield-Addison, Mars Buttfield-Addison, Tim Nugent, Jon Manning
Edition: 1
Publisher: O'Reilly Media
Year: 2022
Language: English
Commentary: Vector PDF
Pages: 332
City: Sebastopol, CA
Tags: Artificial Intelligence; Machine Learning; Reinforcement Learning; Python; Unity; Simulation; PyTorch; Agent-based AI; Imitation Learning; Synthetic Data; Curriculum Learning; Cooperative Learning
Cover
Copyright
Table of Contents
Preface
Resources Used in This Book
Audience and Approach
Organization of This Book
Using This Book
Our Tasks
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Part I. The Basics of Simulation and Synthesis
Chapter 1. Introducing Synthesis and Simulation
A Whole New World of ML
The Domains
Simulation
Synthesis
The Tools
Unity
PyTorch via Unity ML-Agents
Unity ML-Agents Toolkit
Unity Perception
The Techniques
Reinforcement Learning
Imitation Learning
Hybrid Learning
Summary of Techniques
Projects
Simulation Projects
Synthesis Projects
Summary and Next Steps
Chapter 2. Creating Your First Simulation
Everybody Remembers Their First Simulation
Our Simulation
Setting Up
Creating the Unity Project
Packages All the Way Down
The Environment
The Floor
The Target
The Agent
Starting and Stopping the Agent
Letting the Agent Observe the Environment
Letting the Agent Take Actions in the Environment
Giving the Agent Rewards for Its Behavior
Finishing Touches for the Agent
Providing a Manual Control System for the Agent
Training with the Simulation
Monitoring the Training with TensorBoard
When the Training Is Complete
What’s It All Mean?
Coming Up Next
Chapter 3. Creating Your First Synthesized Data
Unity Perception
The Process
Using Unity Perception
Creating the Unity Project
Creating a Scene
Getting the Dice Models
A Very Simple Scene
Preparing for Synthesis
Testing the Scenario
Setting Up Our Labels
Checking the Labels
What’s Next?
Part II. Simulating Worlds for Fun and Profit
Chapter 4. Creating a More Advanced Simulation
Setting Up the Block Pusher
Creating the Unity Project
The Environment
The Floor
The Walls
The Block
The Goal
The Agent
The Environment
Training and Testing
Chapter 5. Creating a Self-Driving Car
Creating the Environment
The Track
The Car
Setting Up for ML
Training the Simulation
Training
When the Training Is Complete
Chapter 6. Introducing Imitation Learning
Simulation Environment
Creating the Ground
Creating the Goal
The Name’s Ball, Agent Ball
The Camera
Building the Simulation
Agent Components
Adding Heuristic Controls
Observations and Goals
Generating Data and Training
Creating Training Data
Configuring for Training
Begin Training
Running with Our Trained Model
Understanding and Using Imitation Learning
Chapter 7. Advanced Imitation Learning
Meet GAIL
Do What I Say and Do
A GAIL Scenario
Modifying the Agent’s Actions
Modifying the Observations
Resetting the Agent
Updating the Agent Properties
Demonstration Time
Training with GAIL
Running It and Beyond
Chapter 8. Introducing Curriculum Learning
Curriculum Learning in ML
A Curriculum Learning Scenario
Building in Unity
Creating the Ground
Creating the Target
The Agent
Building the Simulation
Making the Agent an Agent
Actions
Observations
Heuristic Controls for Humans
Creating the Curriculum
Resetting the Environment
Curriculum Config
Training
Running It
Curriculum Versus Other Approaches
What’s Next?
Chapter 9. Cooperative Learning
A Simulation for Cooperation
Building the Environment in Unity
Coding the Agents
Coding the Environment Manager
Coding the Blocks
Finalizing the Environment and Agents
Training for Cooperation
Cooperative Agents or One Big Agent
Chapter 10. Using Cameras in Simulations
Observations and Camera Sensors
Building a Camera-Only Agent
Coding the Camera-Only Agent
Adding a New Camera for the Agent
Seeing What the Agent’s Camera Sees
Training the Camera-Based Agent
Cameras and You
Chapter 11. Working with Python
Python All the Way Down
Experimenting with an Environment
What Can Be Done with Python?
Using Your Own Environment
Completely Custom Training
What’s the Point of Python?
Chapter 12. Under the Hood and Beyond
Hyperparameters (and Just Parameters)
Parameters
Reward Parameters
Hyperparameters
Algorithms
Unity Inference Engine and Integrations
Using the ML-Agents Gym Wrapper
Side Channels
Part III. Synthetic Data, Real Results
Chapter 13. Creating More Advanced Synthesized Data
Adding Random Elements to the Scene
Randomizing the Floor Color
Randomizing the Camera Position
What’s Next?
Chapter 14. Synthetic Shopping
Creating the Unity Environment
A Perception Camera
Faking It Until You Make It
Using Synthesized Data
Index
About the Authors
Colophon
