Observability Engineering: Achieving Production Excellence

Cover
Copyright
Table of Contents
Foreword
Preface
Who This Book Is For
Why We Wrote This Book
What You Will Learn
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Part I. The Path to Observability
Chapter 1. What Is Observability?
The Mathematical Definition of Observability
Applying Observability to Software Systems
Mischaracterizations About Observability for Software
Why Observability Matters Now
Is This Really the Best Way?
Why Are Metrics and Monitoring Not Enough?
Debugging with Metrics Versus Observability
The Role of Cardinality
The Role of Dimensionality
Debugging with Observability
Observability Is for Modern Systems
Conclusion
Chapter 2. How Debugging Practices Differ Between Observability and Monitoring
How Monitoring Data Is Used for Debugging
Troubleshooting Behaviors When Using Dashboards
The Limitations of Troubleshooting by Intuition
Traditional Monitoring Is Fundamentally Reactive
How Observability Enables Better Debugging
Conclusion
Chapter 3. Lessons from Scaling Without Observability
An Introduction to Parse
Scaling at Parse
The Evolution Toward Modern Systems
The Evolution Toward Modern Practices
Shifting Practices at Parse
Conclusion
Chapter 4. How Observability Relates to DevOps, SRE, and Cloud Native
Cloud Native, DevOps, and SRE in a Nutshell
Observability: Debugging Then Versus Now
Observability Empowers DevOps and SRE Practices
Conclusion
Part II. Fundamentals of Observability
Chapter 5. Structured Events Are the Building Blocks of Observability
Debugging with Structured Events
The Limitations of Metrics as a Building Block
The Limitations of Traditional Logs as a Building Block
Unstructured Logs
Structured Logs
Properties of Events That Are Useful in Debugging
Conclusion
Chapter 6. Stitching Events into Traces
Distributed Tracing and Why It Matters Now
The Components of Tracing
Instrumenting a Trace the Hard Way
Adding Custom Fields into Trace Spans
Stitching Events into Traces
Conclusion
Chapter 7. Instrumentation with OpenTelemetry
A Brief Introduction to Instrumentation
Open Instrumentation Standards
Instrumentation Using Code-Based Examples
Start with Automatic Instrumentation
Add Custom Instrumentation
Send Instrumentation Data to a Backend System
Conclusion
Chapter 8. Analyzing Events to Achieve Observability
Debugging from Known Conditions
Debugging from First Principles
Using the Core Analysis Loop
Automating the Brute-Force Portion of the Core Analysis Loop
This Misleading Promise of AIOps
Conclusion
Chapter 9. How Observability and Monitoring Come Together
Where Monitoring Fits
Where Observability Fits
System Versus Software Considerations
Assessing Your Organizational Needs
Exceptions: Infrastructure Monitoring That Can’t Be Ignored
Real-World Examples
Conclusion
Part III. Observability for Teams
Chapter 10. Applying Observability Practices in Your Team
Join a Community Group
Start with the Biggest Pain Points
Buy Instead of Build
Flesh Out Your Instrumentation Iteratively
Look for Opportunities to Leverage Existing Efforts
Prepare for the Hardest Last Push
Conclusion
Chapter 11. Observability-Driven Development
Test-Driven Development
Observability in the Development Cycle
Determining Where to Debug
Debugging in the Time of Microservices
How Instrumentation Drives Observability
Shifting Observability Left
Using Observability to Speed Up Software Delivery
Conclusion
Chapter 12. Using Service-Level Objectives for Reliability
Traditional Monitoring Approaches Create Dangerous Alert Fatigue
Threshold Alerting Is for Known-Unknowns Only
User Experience Is a North Star
What Is a Service-Level Objective?
Reliable Alerting with SLOs
Changing Culture Toward SLO-Based Alerts: A Case Study
Conclusion
Chapter 13. Acting on and Debugging SLO-Based Alerts
Alerting Before Your Error Budget Is Empty
Framing Time as a Sliding Window
Forecasting to Create a Predictive Burn Alert
The Lookahead Window
The Baseline Window
Acting on SLO Burn Alerts
Using Observability Data for SLOs Versus Time-Series Data
Conclusion
Chapter 14. Observability and the Software Supply Chain
Why Slack Needed Observability
Instrumentation: Shared Client Libraries and Dimensions
Case Studies: Operationalizing the Supply Chain
Understanding Context Through Tooling
Embedding Actionable Alerting
Understanding What Changed
Conclusion
Part IV. Observability at Scale
Chapter 15. Build Versus Buy and Return on Investment
How to Analyze the ROI of Observability
The Real Costs of Building Your Own
The Hidden Costs of Using “Free” Software
The Benefits of Building Your Own
The Risks of Building Your Own
The Real Costs of Buying Software
The Hidden Financial Costs of Commercial Software
The Hidden Nonfinancial Costs of Commercial Software
The Benefits of Buying Commercial Software
The Risks of Buying Commercial Software
Buy Versus Build Is Not a Binary Choice
Conclusion
Chapter 16. Efficient Data Storage
The Functional Requirements for Observability
Time-Series Databases Are Inadequate for Observability
Other Possible Data Stores
Data Storage Strategies
Case Study: The Implementation of Honeycomb’s Retriever
Partitioning Data by Time
Storing Data by Column Within Segments
Performing Query Workloads
Querying for Traces
Querying Data in Real Time
Making It Affordable with Tiering
Making It Fast with Parallelism
Dealing with High Cardinality
Scaling and Durability Strategies
Notes on Building Your Own Efficient Data Store
Conclusion
Chapter 17. Cheap and Accurate Enough: Sampling
Sampling to Refine Your Data Collection
Using Different Approaches to Sampling
Constant-Probability Sampling
Sampling on Recent Traffic Volume
Sampling Based on Event Content (Keys)
Combining per Key and Historical Methods
Choosing Dynamic Sampling Options
When to Make a Sampling Decision for Traces
Translating Sampling Strategies into Code
The Base Case
Fixed-Rate Sampling
Recording the Sample Rate
Consistent Sampling
Target Rate Sampling
Having More Than One Static Sample Rate
Sampling by Key and Target Rate
Sampling with Dynamic Rates on Arbitrarily Many Keys
Putting It All Together: Head and Tail per Key Target Rate Sampling
Conclusion
Chapter 18. Telemetry Management with Pipelines
Attributes of Telemetry Pipelines
Routing
Security and Compliance
Workload Isolation
Data Buffering
Capacity Management
Data Filtering and Augmentation
Data Transformation
Ensuring Data Quality and Consistency
Managing a Telemetry Pipeline: Anatomy
Challenges When Managing a Telemetry Pipeline
Performance
Correctness
Availability
Reliability
Isolation
Data Freshness
Use Case: Telemetry Management at Slack
Metrics Aggregation
Logs and Trace Events
Open Source Alternatives
Managing a Telemetry Pipeline: Build Versus Buy
Conclusion
Part V. Spreading Observability Culture
Chapter 19. The Business Case for Observability
The Reactive Approach to Introducing Change
The Return on Investment of Observability
The Proactive Approach to Introducing Change
Introducing Observability as a Practice
Using the Appropriate Tools
Instrumentation
Data Storage and Analytics
Rolling Out Tools to Your Teams
Knowing When You Have Enough Observability
Conclusion
Chapter 20. Observability’s Stakeholders and Allies
Recognizing Nonengineering Observability Needs
Creating Observability Allies in Practice
Customer Support Teams
Customer Success and Product Teams
Sales and Executive Teams
Using Observability Versus Business Intelligence Tools
Query Execution Time
Accuracy
Recency
Structure
Time Windows
Ephemerality
Using Observability and BI Tools Together in Practice
Conclusion
Chapter 21. An Observability Maturity Model
A Note About Maturity Models
Why Observability Needs a Maturity Model
About the Observability Maturity Model
Capabilities Referenced in the OMM
Respond to System Failure with Resilience
Deliver High-Quality Code
Manage Complexity and Technical Debt
Release on a Predictable Cadence
Understand User Behavior
Using the OMM for Your Organization
Conclusion
Chapter 22. Where to Go from Here
Observability, Then Versus Now
Additional Resources
Predictions for Where Observability Is Going
Index
About the Authors
Colophon