Introduction To Conformal Prediction With Python is the quickest way to learn an easy-to-use and very general technique for uncertainty quantification.
"This concise book is accessible, lucid, and full of helpful code snippets. It explains the mathematical ideas with clarity and provides the reader with practical examples that illustrate the essence of conformal prediction, a powerful idea for uncertainty quantification."
– Junaid Butt, Research Software Engineer, IBM Research
Summary
A prerequisite for trust in machine learning is uncertainty quantification. Without it, an accurate prediction and a wild guess look the same.
Yet many machine learning models come without uncertainty quantification. And while there are many approaches to uncertainty – from Bayesian posteriors to bootstrapping – we have no guarantees that these approaches will perform well on new data.
At first glance conformal prediction seems like yet another contender. But conformal prediction can work in combination with any other uncertainty approach and has many advantages that make it stand out
Guaranteed coverage: Prediction regions generated by conformal prediction come with coverage guarantees of the true outcome
Easy to use: Conformal prediction approaches can be implemented from scratch with just a few lines of code
Model-agnostic: Conformal prediction works with any machine learning model
Distribution-free: Conformal prediction makes no distributional assumptions
No retraining required: Conformal prediction can be used without retraining the model
Broad application: conformal prediction works for classification, regression, time series forecasting, and many other tasks
Sound good?
Author(s): Christoph Molnar
Edition: 1
Publisher: MUCBOOK
Year: 2023
Language: English
Commentary: A Short Guide for Quantifying Uncertainty of Machine Learning Models
Pages: 101
Summary
Preface
Who This Book Is For
Introduction to Conformal Prediction
We need uncertainty quantification
Uncertainty has many sources
Distinguish good from bad predictions
Other approaches don't have guaranteed coverage
Conformal prediction fills the gap
Getting Started with Conformal Prediction in Python
Installing the software
Let's classify some beans
First try: a naive approach
Second try: conformal classification
Getting started with MAPIE
Intuition Behind Conformal Prediction
Conformal prediction is a recipe
Understand parallels to out-of-sample evaluation
How to interpret prediction regions and coverage
Conformal prediction and supervised learning
Classification
Back to the beans
The naive method doesn't work
The Score method is simple but not adaptive
Use Adaptive Prediction Sets (APS) for conditional coverage
Top-k method for fixed size sets
Regularized APS (RAPS) for small sets
Group-balanced conformal prediction
Class-Conditional APS (CCAPS) for coverage by class
Guide for choosing a conformal classification method
Regression and Quantile Regression
Motivation
Rent Index Data
Conformalized Mean Regression
Conformalized Quantile Regression (CQR)
A Glimpse Beyond Classification and Regression
Quickly categorize conformal prediction by task and score
Time Series Forecasting
Multi-Label Classification
Outlier Detection
Probability Calibration
And many more tasks
How to stay up to date
Design Your Own Conformal Predictor
Steps to build your own conformal predictor
Finding the right non-conformity score
Start with a heuristic notion of uncertainty
A general recipe for 1D uncertaity heuristics
Metrics for evaluating conformal predictors
Q & A
How do I choose the calibration size?
How do I make conformal prediction reproducible?
How does alpha affect the size of the prediction regions?
What happens if I choose a large \alpha for conformal classification?
How to interpret empty prediction sets?
Can I use the same data for calibration and model evaluation?
What if I find errors in the book or want to provide feedback?
Acknowledgements
References
