Mastering Reinforcement Learning with Python: Build next-generation, self-learning models using reinforcement learning techniques and best practices

Cover
Title Page
Copyright and Credits
About Packt
Contributors
Table of Contents
Preface
Section 1: Reinforcement Learning Foundations
Chapter 1: Introduction to Reinforcement Learning
Why RL?
The three paradigms of machine learning
Supervised learning
Unsupervised learning
Reinforcement learning
RL application areas and success stories
Games
Robotics and autonomous systems
Supply chain
Manufacturing
Personalization and recommender systems
Smart cities
Elements of an RL problem
RL concepts
Casting tic-tac-toe as an RL problem
Setting up your RL environment
Hardware requirements
Operating system
Software toolbox
Summary
References
Chapter 2: Multi-Armed Bandits
Exploration-exploitation trade-off
What is a MAB?
Problem definition
Experimenting with a simple MAB problem
Case study – online advertising
A/B/n testing
Notation
Application to the online advertising scenario
Advantages and disadvantages of A/B/n testing
ε-greedy actions
Application to the online advertising scenario
Advantages and disadvantages of ε-greedy actions
Action selection using upper confidence bounds
Application to the online advertising scenario
Advantages and disadvantages of using UCBs
Thompson (posterior) sampling
Application to the online advertising scenario
Advantages and disadvantages of Thompson sampling
Summary
References
Chapter 3: Contextual Bandits
Why we need function approximations
Using function approximations for context
Case study – contextual online advertising with synthetic user data
Function approximation with regularized logistic regression
Objective – regret minimization
Solving the online advertising problem
Using function approximations for actions
Case study – contextual online advertising with user data from the US Census
Function approximation using a neural network
Calculating the regret
Solving the online advertising problem
Other applications of multi-armed bandits and CBs
Recommender systems
Web page/app feature design
Healthcare
Dynamic pricing
Finance
Control systems tuning
Summary
References
Chapter 4: Makings of the Markov Decision Process
Starting with Markov chains
Stochastic processes with the Markov property
Classification of states in a Markov chain
Transitionary and steady state behavior
Example – n-step behavior in the grid world
Example – a sample path in an ergodic Markov chain
Semi-Markov processes and continuous-time Markov chains
Introducing the reward – Markov reward process
Attaching rewards to the grid world example
Relationships between average rewards with different initializations
Return, discount, and state values
Analytically calculating the state values
Estimating the state values iteratively
Bringing the action in – MDP
Definition
Grid world as an MDP
State-value function
Action-value function
Optimal state-value and action-value functions
Bellman optimality
Partially observable MDPs
Summary
Exercises
Further reading
Chapter 5: Solving the Reinforcement Learning Problem
Exploring dynamic programming
Example use case – Inventory replenishment of a food truck
Policy evaluation
Policy iteration
Value iteration
Drawbacks of dynamic programming
Training your agent with Monte Carlo methods
Monte Carlo prediction
Monte Carlo control
Temporal-difference learning
One-step TD learning – TD(0)
n-step TD learning
Understanding the importance of simulation in reinforcement learning
Summary
Exercises
References
Section 2: Deep Reinforcement Learning
Chapter 6: Deep Q-Learning at Scale
From tabular Q-learning to deep Q-learning
Neural-fitted Q-iteration
Online Q-learning
Deep Q-networks
Key concepts in DQNs
The DQN algorithm
Extensions to the DQN – Rainbow
The extensions
The performance of the integrated agent
How to choose which extensions to use – ablations to Rainbow
What happened to the deadly triad?
Distributed deep Q-learning
Components of a distributed deep Q-learning architecture
Gorila – general RL architecture
Ape-X – distributed prioritized experience replay
Implementing scalable deep Q-learning algorithms using Ray
A primer on Ray
Ray implementation of a DQN variant
Using RLlib for production-grade deep RL
Summary
References
Chapter 7: Policy-Based Methods
Why should we use policy-based methods?
A more principled approach
The ability to use continuous action spaces
The ability to learn truly random stochastic policies
The vanilla policy gradient
The objective in policy gradient methods
Figuring out the gradient
REINFORCE
The problem with REINFORCE and all policy gradient methods
Vanilla policy gradient using RLlib
Actor-critic methods
Further reducing the variance in policy-based methods
Advantage Actor-Critic – A2C
Asynchronous Advantage Actor-Critic: A3C
Generalized advantage estimators
Trust-region methods
Policy gradient as policy iteration
TRPO – Trust Region Policy Optimization
PPO – Proximal Policy Optimization
Off-policy methods
DDPG – Deep Deterministic Policy Gradient
TD3 – Twin Delayed Deep Deterministic Policy Gradient
SAC – Soft actor-critic
IMPALA – Importance Weighted Actor-Learner Architecture
A comparison of the policy-based methods in Lunar Lander
How to pick the right algorithm
Open source implementations of policy-gradient methods
Summary
References
Chapter 8: Model-Based Methods
Technical requirements
Introducing model-based methods
Planning through a model
Defining the optimal control problem
Random shooting
Cross-entropy method
Covariance matrix adaptation evolution strategy
Monte Carlo tree search
Learning a world model
Understanding what model means
Identifying when to learn a model
Introducing a general procedure to learn a model
Understanding and mitigating the impact of model uncertainty
Learning a model from complex observations
Unifying model-based and model-free approaches
Refresher on Q-learning
Dyna-style acceleration of model-free methods using world models
Summary
References
Chapter 9: Multi-Agent Reinforcement Learning
Introducing multi-agent reinforcement learning
Collaboration and competition between MARL agents
Exploring the challenges in multi-agent reinforcement learning
Non-stationarity
Scalability
Unclear reinforcement learning objective
Information sharing
Training policies in multi-agent settings
RLlib multi-agent environment
Competitive self-play
Training tic-tac-toe agents through self-play
Designing the multi-agent tic-tac-toe environment
Configuring the trainer
Observing the results
Summary
References
Section 3: Advanced Topics in RL
Chapter 10: Machine Teaching
Technical requirements
Introduction to MT
Understanding the need for MT
Exploring the elements of MT
Engineering the reward function
When to engineer the reward function
Reward shaping
Example – reward shaping for a mountain car
Challenges with engineering the reward function
Curriculum learning
Warm starts and demonstration learning
Action masking
Concept networks
Downsides and the promises of MT
Summary
References
Chapter 11: Generalization and Domain Randomization
An overview of generalization and partial observability
Generalization and overfitting in supervised learning
Generalization and overfitting in RL
The connection between generalization and partial observability
Overcoming partial observability with memory
Overcoming overfitting with randomization
Recipe for generalization
Domain randomization for generalization
Dimensions of randomization
Quantifying generalization
The effect of regularization and network architecture on the generalization of RL policies
Network randomization and feature matching
Curriculum learning for generalization
Sunblaze environment
Using memory to overcome partial observability
Stacking observations
Using RNNs
Transformer architecture
Summary
References
Chapter 12: Meta-Reinforcement Learning
Introduction to meta-RL
Learning to learn
Defining meta-RL
Relation to animal learning and the Harlow experiment
Relation to partial observability and domain randomization
Meta-RL with recurrent policies
Grid world example
RLlib implementation
Gradient-based meta-RL
RLlib implementation
Meta-RL as partially observed RL
Challenges in meta-RL
Summary
References
Chapter 13: Other Advanced Topics
Distributed reinforcement learning
Scalable, efficient deep reinforcement learning – SEED RL
Recurrent experience replay in distributed reinforcement learning
Experimenting with SEED RL and R2D2
Curiosity-driven reinforcement learning
Curiosity-driven learning for hard-exploration problems
Challenges in curiosity-driven reinforcement learning
Never Give Up
Agent57 improvements
Offline reinforcement learning
An overview of how offline reinforcement learning works
Why we need special algorithms for offline learning
Why offline reinforcement learning is crucial
Advantage weighted actor-critic
Offline reinforcement learning benchmarks
Summary
References
Section 4: Applications of RL
Chapter 14: Autonomous Systems
Introducing PyBullet
Setting up PyBullet
Getting familiar with the KUKA environment
Grasping a rectangular block using a KUKA robot
The KUKA Gym environment
Developing strategies to solve the KUKA environment
Parametrizing the difficulty of the problem
Using curriculum learning to train the KUKA robot
Customizing the environment for curriculum learning
Designing the lessons in the curriculum
Training the agent using a manually designed curriculum
Curriculum learning using absolute learning progress
Comparing the experiment results
Going beyond PyBullet into autonomous driving
Summary
References
Chapter 15: Supply Chain Management
Optimizing inventory procurement decisions
The need for inventory and the trade-offs in its management
Components of an inventory optimization problem
Single-step inventory optimization – the newsvendor problem
Simulating multi-step inventory dynamics
Developing a near-optimal benchmark policy
A reinforcement learning solution for inventory management
Modeling routing problems
Pick-up and delivery of online meal orders
Pointer networks for dynamic combinatorial optimization
Summary
References
Chapter 16: Marketing, Personalization and Finance
Going beyond bandits for personalization
Shortcomings of bandit models
Deep RL for news recommendation
Developing effective marketing strategies using RL
Personalized marketing content
Marketing resource allocation for customer acquisition
Reducing the customer churn rate
Winning back lost customers
Applying RL in finance
Challenges with using RL in finance
Introducing TensorTrade
Developing equity trading strategies
Summary
References
Chapter 17: Smart City and Cybersecurity
Traffic light control to optimize vehicle flow
Introducing Flow
Creating an experiment in Flow
Modeling the traffic light control problem
Solving the traffic control problem using RLlib
Further reading
Providing an ancillary service to a power grid
Power grid operations and ancillary services
Describing the environment and the decision-making problem
RL model
Detecting cyberattacks in a smart grid
The problem of early detection of cyberattacks in a power grid
Partial observability of the grid state
Summary
References
Chapter 18: Challenges and Future Directions in Reinforcement Learning
What you have achieved with this book
Challenges and future directions
Sample efficiency
Need for high-fidelity and fast simulation models
High-dimensional action spaces
Reward function fidelity
Safety, behavior guarantees, and explainability
Reproducibility and sensitivity to hyperparameter choices
Robustness and adversarial agents
Suggestions for aspiring RL experts
Go deeper into the theory
Follow good practitioners and research labs
Learn from papers and their good explanations
Stay up to date with trends in other fields of deep learning
Read open source repos
Practice!
Final words
References
Other Books You May Enjoy
Index