Sequential decision problems arise in virtually every human process. They span finance, energy, transportation, health, e-commerce, and supply chains and include pure learning problems that arise in laboratory or field experiments. They even cover search algorithms to maximize uncertain functions. An important dimension of every problem setting is the need to make decisions in the presence of different forms of uncertainty and evolving information processes.
Warren B. Powell's work in sequential decision problems started in the 1980s and spanned rail, energy, health, finance, e-commerce, supply chain management, and even learning for materials science. His work on a wide range of problems highlighted the importance of using a variety of methods. In the process, he came to realize that any sequential decision problem can be modeled using a single universal framework that involves searching over methods for making decisions.
The goal of this book is to enable readers to understand how to approach, model and solve a sequential decision problem. To that end, it uses a teach-by-example style to illustrate a modeling framework that can represent any sequential decision problem. It tackles the challenge of designing methods, called policies, for making decisions and describes four classes of policies that are universal in that they span any method that might be used; whether from the academic literature or heuristics used in practice. While this does not mean that every problem can be solved immediately, the framework helps avoid the tendency in the academic literature of focusing on narrow classes of methods.
Author(s): Warren B. Powell
Publisher: Now Publishers Inc
Year: 2022
Language: English
Pages: 316
Preface
Modeling Sequential Decision Problems
Getting started
The modeling process
Some inventory problems
The mathematical modeling framework
Modeling uncertainty
Designing policies
Next steps
What did we learn?
Exercises
An Asset Selling Problem
Narrative
Basic model
Modeling uncertainty
Designing policies
Policy evaluation
Extensions
What did we learn?
Exercises
Adaptive Market Planning
Narrative
Basic model
Uncertainty modeling
Designing policies
Extensions
What did we learn?
Exercises
Learning the Best Diabetes Medication
Narrative
Basic model
Modeling uncertainty
Designing policies
Policy evaluation
Extensions
What did we learn?
Exercises
Stochastic Shortest Path Problems - Static
Narrative
Basic model
Modeling uncertainty
Designing policies
Policy evaluation
Extension - Adaptive stochastic shortest paths
What did we learn?
Exercises
Stochastic Shortest Path Problems - Dynamic
Narrative
Basic model
Modeling uncertainty
Designing policies
What did we learn?
Exercises
Applications, Revisited
The four classes of policies
Models, revisited
On the suboptimality of direct lookahead policies
Online vs. offline objectives
Derivative-based policy search
Derivative-free policy search
What did we learn?
Exercises
Energy Storage I
Narrative
Basic model
Modeling uncertainty
Designing policies
What did we learn?
Exercises
Energy Storage II
Narrative
Basic model
Modeling uncertainty
Designing policies
What did we learn?
Exercises
Supply Chain Management I: The Two-agent Newsvendor Problem
Narrative
Basic model
Modeling uncertainty
Designing policies
What did we learn?
Exercises
Supply Chain Management II: The Beer Game
Narrative
Basic model
Modeling uncertainty
Designing policies
Extensions
What did we learn?
Exercises
Ad-click Optimization
Narrative
Basic model
Modeling uncertainty
Designing policies
Extension: Customers with simple attributes
What did we learn?
Exercises
Blood Management Problem
Narrative
Basic model
Modeling uncertainty
Designing policies
Extensions
What did we learn?
Exercises
Optimizing Clinical Trials
Narrative
Basic model
Modeling uncertainty
Designing policies
What did we learn?
Exercises
Acknowledgments
References
