Foundations of Reinforcement Learning with Applications in Finance

Cover
Half Title
Title Page
Copyright Page
Contents
Preface
Author Biographies
Summary of Notation
CHAPTER 1: Overview
1.1. LEARNING REINFORCEMENT LEARNING
1.2. WHAT YOU WILL LEARN FROM THIS BOOK
1.3. EXPECTED BACKGROUND TO READ THIS BOOK
1.4. DECLUTTERING THE JARGON LINKED TO REINFORCEMENT LEARNING
1.5. INTRODUCTION TO THE MARKOV DECISION PROCESS (MDP) FRAMEWORK
1.6. REAL-WORLD PROBLEMS THAT FIT THE MDP FRAMEWORK
1.7. THE INHERENT DIFFICULTY IN SOLVING MDPS
1.8. VALUE FUNCTION, BELLMAN EQUATIONS, DYNAMIC PROGRAMMING AND RL
1.9. OUTLINE OF CHAPTERS
1.9.1. Module I: Processes and Planning Algorithms
1.9.2. Module II: Modeling Financial Applications
1.9.3. Module III: Reinforcement Learning Algorithms
1.9.4. Module IV: Finishing Touches
1.9.5. Short Appendix Chapters
CHAPTER 2: Programming and Design
2.1. CODE DESIGN
2.2. ENVIRONMENT SETUP
2.3. CLASSES AND INTERFACES
2.3.1. A Distribution Interface
2.3.2. A Concrete Distribution
2.3.2.1. Dataclasses
2.3.2.2. Immutability
2.3.3. Checking Types
2.3.3.1. Static Typing
2.3.4. Type Variables
2.3.5. Functionality
2.4. ABSTRACTING OVER COMPUTATION
2.4.1. First-Class Functions
2.4.1.1. Lambdas
2.4.2. Iterative Algorithms
2.4.2.1. Iterators and Generators
2.5. KEY TAKEAWAYS FROM THIS CHAPTER
MODULE I: Processes and Planning Algorithms
Chapter 3: Markov Processes
3.1. THE CONCEPT OF STATE IN A PROCESS
3.2. UNDERSTANDING MARKOV PROPERTY FROM STOCK PRICE EXAMPLES
3.3. FORMAL DEFINITIONS FOR MARKOV PROCESSES
3.3.1. Starting States
3.3.2. Terminal States
3.3.3. Markov Process Implementation
3.4. STOCK PRICE EXAMPLES MODELED AS MARKOV PROCESSES
3.5. FINITE MARKOV PROCESSES
3.6. SIMPLE INVENTORY EXAMPLE
3.7. STATIONARY DISTRIBUTION OF A MARKOV PROCESS
3.8. FORMALISM OF MARKOV REWARD PROCESSES
3.9. SIMPLE INVENTORY EXAMPLE AS A MARKOV REWARD PROCESS
3.10. FINITE MARKOV REWARD PROCESSES
3.11. SIMPLE INVENTORY EXAMPLE AS A FINITE MARKOV REWARD PROCESS
3.12. VALUE FUNCTION OF A MARKOV REWARD PROCESS
3.13. SUMMARY OF KEY LEARNINGS FROM THIS CHAPTER
Chapter 4: Markov Decision Processes
4.1. SIMPLE INVENTORY EXAMPLE: HOW MUCH TO ORDER?
4.2. THE DIFFICULTY OF SEQUENTIAL DECISIONING UNDER UNCERTAINTY
4.3. FORMAL DEFINITION OF A MARKOV DECISION PROCESS
4.4. POLICY
4.5. [MARKOV DECISION PROCESS, POLICY] := MARKOV REWARD PROCESS
4.6. SIMPLE INVENTORY EXAMPLE WITH UNLIMITED CAPACITY (INFINITE STATE/ACTION SPACE)
4.7. FINITE MARKOV DECISION PROCESSES
4.8. SIMPLE INVENTORY EXAMPLE AS A FINITE MARKOV DECISION PROCESS
4.9. MDP VALUE FUNCTION FOR A FIXED POLICY
4.10. OPTIMAL VALUE FUNCTION AND OPTIMAL POLICIES
4.11. VARIANTS AND EXTENSIONS OF MDPS
4.11.1. Size of Spaces and Discrete versus Continuous
4.11.1.1. State Space
4.11.1.2. Action Space
4.11.1.3. Time Steps
4.11.2. Partially-Observable Markov Decision Processes (POMDPs)
4.12. SUMMARY OF KEY LEARNINGS FROM THIS CHAPTER
Chapter 5: Dynamic Programming Algorithms
5.1. PLANNING VERSUS LEARNING
5.2. USAGE OF THE TERM DYNAMIC PROGRAMMING
5.3. FIXED-POINT THEORY
5.4. BELLMAN POLICY OPERATOR AND POLICY EVALUATION ALGORITHM
5.5. GREEDY POLICY
5.6. POLICY IMPROVEMENT
5.7. POLICY ITERATION ALGORITHM
5.8. BELLMAN OPTIMALITY OPERATOR AND VALUE ITERATION ALGORITHM
5.9. OPTIMAL POLICY FROM OPTIMAL VALUE FUNCTION
5.10. REVISITING THE SIMPLE INVENTORY EXAMPLE
5.11. GENERALIZED POLICY ITERATION
5.12. ASYNCHRONOUS DYNAMIC PROGRAMMING
5.13. FINITE-HORIZON DYNAMIC PROGRAMMING: BACKWARD INDUCTION
5.14. DYNAMIC PRICING FOR END-OF-LIFE/END-OF-SEASON OF A PRODUCT
5.15. GENERALIZATION TO NON-TABULAR ALGORITHMS
5.16. SUMMARY OF KEY LEARNINGS FROM THIS CHAPTER
Chapter 6: Function Approximation and Approximate Dynamic Programming
6.1. FUNCTION APPROXIMATION
6.2. LINEAR FUNCTION APPROXIMATION
6.3. NEURAL NETWORK FUNCTION APPROXIMATION
6.4. TABULAR AS A FORM OF FUNCTIONAPPROX
6.5. APPROXIMATE POLICY EVALUATION
6.6. APPROXIMATE VALUE ITERATION
6.7. FINITE-HORIZON APPROXIMATE POLICY EVALUATION
6.8. FINITE-HORIZON APPROXIMATE VALUE ITERATION
6.9. FINITE-HORIZON APPROXIMATE Q-VALUE ITERATION
6.10. HOW TO CONSTRUCT THE NON-TERMINAL STATES DISTRIBUTION
6.11. KEY TAKEAWAYS FROM THIS CHAPTER
MODULE II: Modeling Financial Applications
Chapter 7: Utility Theory
7.1. INTRODUCTION TO THE CONCEPT OF UTILITY
7.2. A SIMPLE FINANCIAL EXAMPLE
7.3. THE SHAPE OF THE UTILITY FUNCTION
7.4. CALCULATING THE RISK-PREMIUM
7.5. CONSTANT ABSOLUTE RISK-AVERSION (CARA)
7.6. A PORTFOLIO APPLICATION OF CARA
7.7. CONSTANT RELATIVE RISK-AVERSION (CRRA)
7.8. A PORTFOLIO APPLICATION OF CRRA
7.9. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 8: Dynamic Asset-Allocation and Consumption
8.1. OPTIMIZATION OF PERSONAL FINANCE
8.2. MERTON’S PORTFOLIO PROBLEM AND SOLUTION
8.3. DEVELOPING INTUITION FOR THE SOLUTION TO MERTON’S PORTFOLIO PROBLEM
8.4. A DISCRETE-TIME ASSET-ALLOCATION EXAMPLE
8.5. PORTING TO REAL-WORLD
8.6. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 9: Derivatives Pricing and Hedging
9.1. A BRIEF INTRODUCTION TO DERIVATIVES
9.1.1. Forwards
9.1.2. European Options
9.1.3. American Options
9.2. NOTATION FOR THE SINGLE-PERIOD SIMPLE SETTING
9.3. PORTFOLIOS, ARBITRAGE AND RISK-NEUTRAL PROBABILITY MEASURE
9.4. FIRST FUNDAMENTAL THEOREM OF ASSET PRICING (1ST FTAP)
9.5. SECOND FUNDAMENTAL THEOREM OF ASSET PRICING (2ND FTAP)
9.6. DERIVATIVES PRICING IN SINGLE-PERIOD SETTING
9.6.1. Derivatives Pricing When Market Is Complete
9.6.2. Derivatives Pricing When Market Is Incomplete
9.6.3. Derivatives Pricing When Market Has Arbitrage
9.7. DERIVATIVES PRICING IN MULTI-PERIOD/CONTINUOUS-TIME
9.7.1. Multi-Period Complete-Market Setting
9.7.2. Continuous-Time Complete-Market Setting
9.8. OPTIMAL EXERCISE OF AMERICAN OPTIONS CAST AS A FINITE MDP
9.9. GENERALIZING TO OPTIMAL-STOPPING PROBLEMS
9.10. PRICING/HEDGING IN AN INCOMPLETE MARKET CAST AS AN MDP
9.11. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 10: Order-Book Trading Algorithms
10.1. BASICS OF ORDER BOOK AND PRICE IMPACT
10.2. OPTIMAL EXECUTION OF A MARKET ORDER
10.2.1. Simple Linear Price Impact Model with No Risk-Aversion
10.2.2. Paper by Bertsimas and Lo on Optimal Order Execution
10.2.3. Incorporating Risk-Aversion and Real-World Considerations
10.3. OPTIMAL MARKET-MAKING
10.3.1. Avellaneda-Stoikov Continuous-Time Formulation
10.3.2. Solving the Avellaneda-Stoikov Formulation
10.3.3. Analytical Approximation to the Solution to Avellaneda-Stoikov Formulation
10.3.4. Real-World Market-Making
10.4. KEY TAKEAWAYS FROM THIS CHAPTER
MODULE III: Reinforcement Learning Algorithms
Chapter 11: Monte-Carlo and Temporal-Difference for Prediction
11.1. OVERVIEW OF THE REINFORCEMENT LEARNING APPROACH
11.2. RL FOR PREDICTION
11.3. MONTE-CARLO (MC) PREDICTION
11.4. TEMPORAL-DIFFERENCE (TD) PREDICTION
11.5. TD VERSUS MC
11.5.1. TD Learning Akin to Human Learning
11.5.2. Bias, Variance and Convergence
11.5.3. Fixed-Data Experience Replay on TD versus MC
11.5.4. Bootstrapping and Experiencing
11.6. TD(λ) PREDICTION
11.6.1. n-Step Bootstrapping Prediction Algorithm
11.6.2. λ-Return Prediction Algorithm
11.6.3. Eligibility Traces
11.6.4. Implementation of the TD(λ) Prediction Algorithm
11.7. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 12: Monte-Carlo and Temporal-Difference for Control
12.1. REFRESHER ON GENERALIZED POLICY ITERATION (GPI)
12.2. GPI WITH EVALUATION AS MONTE-CARLO
12.3. GLIE MONTE-CONTROL CONTROL
12.4. SARSA
12.5. SARSA(λ)
12.6. OFF-POLICY CONTROL
12.6.1. Q-Learning
12.6.2. Windy Grid
12.6.3. Importance Sampling
12.7. CONCEPTUAL LINKAGE BETWEEN DP AND TD ALGORITHMS
12.8. CONVERGENCE OF RL ALGORITHMS
12.9. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 13: Batch RL, Experience-Replay, DQN, LSPI, Gradient TD
13.1. BATCH RL AND EXPERIENCE-REPLAY
13.2. A GENERIC IMPLEMENTATION OF EXPERIENCE-REPLAY
13.3. LEAST-SQUARES RL PREDICTION
13.3.1. Least-Squares Monte-Carlo (LSMC)
13.3.2. Least-Squares Temporal-Difference (LSTD)
13.3.3. LSTD(λ)
13.3.4. Convergence of Least-Squares Prediction
13.4. Q-LEARNING WITH EXPERIENCE-REPLAY
13.4.1. Deep Q-Networks (DQN) Algorithm
13.5. LEAST-SQUARES POLICY ITERATION (LSPI)
13.5.1. Saving Your Village from a Vampire
13.5.2. Least-Squares Control Convergence
13.6. RL FOR OPTIMAL EXERCISE OF AMERICAN OPTIONS
13.6.1. LSPI for American Options Pricing
13.6.2. Deep Q-Learning for American Options Pricing
13.7. VALUE FUNCTION GEOMETRY
13.7.1. Notation and Definitions
13.7.2. Bellman Policy Operator and Projection Operator
13.7.3. Vectors of Interest in the Φ Subspace
13.8. GRADIENT TEMPORAL-DIFFERENCE (GRADIENT TD)
13.9. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 14: Policy Gradient Algorithms
14.1. ADVANTAGES AND DISADVANTAGES OF POLICY GRADIENT ALGORITHMS
14.2. POLICY GRADIENT THEOREM
14.2.1. Notation and Definitions
14.2.2. Statement of the Policy Gradient Theorem
14.2.3. Proof of the Policy Gradient Theorem
14.3. SCORE FUNCTION FOR CANONICAL POLICY FUNCTIONS
14.3.1. Canonical π(s, a; θ) for Finite Action Spaces
14.3.2. Canonical π(s, a; θ) for Single-Dimensional Continuous Action Spaces
14.4. REINFORCE ALGORITHM (MONTE-CARLO POLICY GRADIENT)
14.5. OPTIMAL ASSET ALLOCATION (REVISITED)
14.6. ACTOR-CRITIC AND VARIANCE REDUCTION
14.7. OVERCOMING BIAS WITH COMPATIBLE FUNCTION APPROXIMATION
14.8. POLICY GRADIENT METHODS IN PRACTICE
14.8.1. Natural Policy Gradient
14.8.2. Deterministic Policy Gradient
14.9. EVOLUTIONARY STRATEGIES
14.10. KEY TAKEAWAYS FROM THIS CHAPTER
MODULE IV: Finishing Touches
Chapter 15: Multi-Armed Bandits: Exploration versus Exploitation
15.1. INTRODUCTION TO THE MULTI-ARMED BANDIT PROBLEM
15.1.1. Some Examples of Explore-Exploit Dilemma
15.1.2. Problem Definition
15.1.3. Regret
15.1.4. Counts and Gaps
15.2. SIMPLE ALGORITHMS
15.2.1. Greedy and ϵ-Greedy
15.2.2. Optimistic Initialization
15.2.3. Decaying ϵt-Greedy Algorithm
15.3. LOWER BOUND
15.4. UPPER CONFIDENCE BOUND ALGORITHMS
15.4.1. Hoeffding’s Inequality
15.4.2. UCB1 Algorithm
15.4.3. Bayesian UCB
15.5. PROBABILITY MATCHING
15.5.1. Thompson Sampling
15.6. GRADIENT BANDITS
15.7. HORSE RACE
15.8. INFORMATION STATE SPACE MDP
15.9. EXTENDING TO CONTEXTUAL BANDITS AND RL CONTROL
15.10. KEY TAKEAWAYS FROM THIS CHAPTER
Chapter 16: Blending Learning and Planning
16.1. PLANNING VERSUS LEARNING
16.1.1. Planning the Solution of Prediction/Control
16.1.2. Learning the Solution of Prediction/Control
16.1.3. Advantages and Disadvantages of Planning versus Learning
16.1.4. Blending Planning and Learning
16.2. DECISION-TIME PLANNING
16.3. MONTE-CARLO TREE-SEARCH (MCTS)
16.4. ADAPTIVE MULTI-STAGE SAMPLING
16.5. SUMMARY OF KEY LEARNINGS FROM THIS CHAPTER
Chapter 17: Summary and Real-World Considerations
17.1. SUMMARY OF KEY LEARNINGS FROM THIS BOOK
17.2. RL IN THE REAL-WORLD
APPENDIX A: Moment Generating Function and Its Applications
A.1. THE MOMENT GENERATING FUNCTION (MGF)
A.2. MGF FOR LINEAR FUNCTIONS OF RANDOM VARIABLES
A.3. MGF FOR THE NORMAL DISTRIBUTION
A.4. MINIMIZING THE MGF
A.4.1. Minimizing the MGF When x Follows a Normal Distribution
A.4.2. Minimizing the MGF When x Follows a Symmetric Binary Distribution
APPENDIX B: Portfolio Theory
B.1. SETTING AND NOTATION
B.2. PORTFOLIO RETURNS
B.3. DERIVATION OF EFFICIENT FRONTIER CURVE
B.4. GLOBAL MINIMUM VARIANCE PORTFOLIO (GMVP)
B.5. ORTHOGONAL EFFICIENT PORTFOLIOS
B.6. TWO-FUND THEOREM
B.7. AN EXAMPLE OF THE EFFICIENT FRONTIER FOR 16 ASSETS
B.8. CAPM: LINEARITY OF COVARIANCE VECTOR W.R.T. MEAN RETURNS
B.9. USEFUL COROLLARIES OF CAPM
B.10. CROSS-SECTIONAL VARIANCE
B.11. EFFICIENT SET WITH A RISK-FREE ASSET
APPENDIX C: Introduction to and Overview of Stochastic Calculus Basics
C.1. SIMPLE RANDOM WALK
C.2. BROWNIAN MOTION AS SCALED RANDOM WALK
C.3. CONTINUOUS-TIME STOCHASTIC PROCESSES
C.4. PROPERTIES OF BROWNIAN MOTION SAMPLE TRACES
C.5. ITO INTEGRAL
C.6. ITO’S LEMMA
C.7. A LOGNORMAL PROCESS
C.8. A MEAN-REVERTING PROCESS
APPENDIX D: The Hamilton-Jacobi-Bellman (HJB) Equation
D.1. HJB AS A CONTINUOUS-TIME VERSION OF BELLMAN OPTIMALITY EQUATION
D.2. HJB WITH STATE TRANSITIONS AS AN ITO PROCESS
APPENDIX E: Black-Scholes Equation and Its Solution for Call/Put Options
E.1. ASSUMPTIONS
E.2. DERIVATION OF THE BLACK-SCHOLES EQUATION
E.3. SOLUTION OF THE BLACK-SCHOLES EQUATION FOR CALL/PUT OPTIONS
APPENDIX F: Function Approximations as Affine Spaces
F.1. VECTOR SPACE
F.2. FUNCTION SPACE
F.3. LINEAR MAP OF VECTOR SPACES
F.4. AFFINE SPACE
F.5. AFFINE MAP
F.6. FUNCTION APPROXIMATIONS
F.6.1. D[R] as an Affine Space P
F.6.2. Delegator Space R
F.7. STOCHASTIC GRADIENT DESCENT
F.8. SGD UPDATE FOR LINEAR FUNCTION APPROXIMATIONS
APPENDIX G: Conjugate Priors for Gaussian and Bernoulli Distributions
G.1. CONJUGATE PRIOR FOR GAUSSIAN DISTRIBUTION
G.2. CONJUGATE PRIOR FOR BERNOULLI DISTRIBUTION
Bibliography
Index