More than half of the analytics and machine learning (ML) models created by organizations today never make it into production. Some of the challenges and barriers to operationalization are technical, but others are organizational. Either way, the bottom line is that models not in production can't provide business impact.
This book introduces the key concepts of MLOps to help data scientists and application engineers not only operationalize ML models to drive real business change but also maintain and improve those models over time. Through lessons based on numerous MLOps applications around the world, nine experts in machine learning provide insights into the five steps of the model life cycle--Build, Preproduction, Deployment, Monitoring, and Governance--uncovering how robust MLOps processes can be infused throughout.
This book helps you:
• Fulfill data science value by reducing friction throughout ML pipelines and workflows
• Refine ML models through retraining, periodic tuning, and complete remodeling to ensure long-term accuracy
• Design the MLOps life cycle to minimize organizational risks with models that are unbiased, fair, and explainable
• Operationalize ML models for pipeline deployment and for external business systems that are more complex and less standardized
Author(s): Mark Treveil, Nicolas Omont, Clément Stenac, Kenji Lefevre, Du Phan, Joachim Zentici, Adrien Lavoillotte, Makoto Miyazaki, Lynn Heidmann
Edition: 1
Publisher: O'Reilly Media
Year: 2020
Language: English
Commentary: Vector PDF
Pages: 186
City: Sebastopol, CA
Tags: Machine Learning; Data Science; Recommender Systems; Marketing; Feature Engineering; Monitoring; Logging; Pipelines; Scalability; Risk Assessment; Finance; Best Practices; Forecasting; Continuous Delivery; Continuous Integration; Containerization; MLOps; Data Engineering; Workflows; Model Deployment; Feedback Loops
Copyright
Table of Contents
Preface
Who This Book Is For
How This Book Is Organized
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Part I. MLOps: What and Why
Chapter 1. Why Now and Challenges
Defining MLOps and Its Challenges
MLOps to Mitigate Risk
Risk Assessment
Risk Mitigation
MLOps for Responsible AI
MLOps for Scale
Closing Thoughts
Chapter 2. People of MLOps
Subject Matter Experts
Data Scientists
Data Engineers
Software Engineers
DevOps
Model Risk Manager/Auditor
Machine Learning Architect
Closing Thoughts
Chapter 3. Key MLOps Features
A Primer on Machine Learning
Model Development
Establishing Business Objectives
Data Sources and Exploratory Data Analysis
Feature Engineering and Selection
Training and Evaluation
Reproducibility
Responsible AI
Productionalization and Deployment
Model Deployment Types and Contents
Model Deployment Requirements
Monitoring
DevOps Concerns
Data Scientist Concerns
Business Concerns
Iteration and Life Cycle
Iteration
The Feedback Loop
Governance
Data Governance
Process Governance
Closing Thoughts
Part II. MLOps: How
Chapter 4. Developing Models
What Is a Machine Learning Model?
In Theory
In Practice
Required Components
Different ML Algorithms, Different MLOps Challenges
Data Exploration
Feature Engineering and Selection
Feature Engineering Techniques
How Feature Selection Impacts MLOps Strategy
Experimentation
Evaluating and Comparing Models
Choosing Evaluation Metrics
Cross-Checking Model Behavior
Impact of Responsible AI on Modeling
Version Management and Reproducibility
Closing Thoughts
Chapter 5. Preparing for Production
Runtime Environments
Adaptation from Development to Production Environments
Data Access Before Validation and Launch to Production
Final Thoughts on Runtime Environments
Model Risk Evaluation
The Purpose of Model Validation
The Origins of ML Model Risk
Quality Assurance for Machine Learning
Key Testing Considerations
Reproducibility and Auditability
Machine Learning Security
Adversarial Attacks
Other Vulnerabilities
Model Risk Mitigation
Changing Environments
Interactions Between Models
Model Misbehavior
Closing Thoughts
Chapter 6. Deploying to Production
CI/CD Pipelines
Building ML Artifacts
What’s in an ML Artifact?
The Testing Pipeline
Deployment Strategies
Categories of Model Deployment
Considerations When Sending Models to Production
Maintenance in Production
Containerization
Scaling Deployments
Requirements and Challenges
Closing Thoughts
Chapter 7. Monitoring and Feedback Loop
How Often Should Models Be Retrained?
Understanding Model Degradation
Ground Truth Evaluation
Input Drift Detection
Drift Detection in Practice
Example Causes of Data Drift
Input Drift Detection Techniques
The Feedback Loop
Logging
Model Evaluation
Online Evaluation
Closing Thoughts
Chapter 8. Model Governance
Who Decides What Governance the Organization Needs?
Matching Governance with Risk Level
Current Regulations Driving MLOps Governance
Pharmaceutical Regulation in the US: GxP
Financial Model Risk Management Regulation
GDPR and CCPA Data Privacy Regulations
The New Wave of AI-Specific Regulations
The Emergence of Responsible AI
Key Elements of Responsible AI
Element 1: Data
Element 2: Bias
Element 3: Inclusiveness
Element 4: Model Management at Scale
Element 5: Governance
A Template for MLOps Governance
Step 1: Understand and Classify the Analytics Use Cases
Step 2: Establish an Ethical Position
Step 3: Establish Responsibilities
Step 4: Determine Governance Policies
Step 5: Integrate Policies into the MLOps Process
Step 6: Select the Tools for Centralized Governance Management
Step 7: Engage and Educate
Step 8: Monitor and Refine
Closing Thoughts
Part III. MLOps: Real-World Examples
Chapter 9. MLOps in Practice: Consumer Credit Risk Management
Background: The Business Use Case
Model Development
Model Bias Considerations
Prepare for Production
Deploy to Production
Closing Thoughts
Chapter 10. MLOps in Practice: Marketing Recommendation Engines
The Rise of Recommendation Engines
The Role of Machine Learning
Push or Pull?
Data Preparation
Design and Manage Experiments
Model Training and Deployment
Scalability and Customizability
Monitoring and Retraining Strategy
Real-Time Scoring
Ability to Turn Recommendations On and Off
Pipeline Structure and Deployment Strategy
Monitoring and Feedback
Retraining Models
Updating Models
Runs Overnight, Sleeps During Daytime
Option to Manually Control Models
Option to Automatically Control Models
Monitoring Performance
Closing Thoughts
Chapter 11. MLOps in Practice: Consumption Forecast
Power Systems
Data Collection
Problem Definition: Machine Learning, or Not Machine Learning?
Spatial and Temporal Resolution
Implementation
Modeling
Deployment
Monitoring
Closing Thoughts
Index
About the Authors
Colophon
