Mastering Distributed Tracing: Analyzing performance in microservices and complex systems

Cover
Copyright
Mapt upsell
Contributors
Table of Contents
Preface
Part I: Introduction
Chapter 1: Why Distributed Tracing?
Microservices and cloud-native applications
What is observability?
The observability challenge of microservices
Traditional monitoring tools
Metrics
Logs
Distributed tracing
My experience with tracing
Why this book?
Summary
References
Chapter 2: Take Tracing for a HotROD Ride
Prerequisites
Running from prepackaged binaries
Running from Docker images
Running from the source code
Go language development environment
Jaeger source code
Start Jaeger
Meet the HotROD
The architecture
The data flow
Contextualized logs
Span tags versus logs
Identifying sources of latency
Resource usage attribution
Summary
References
Chapter 3: Distributed Tracing Fundamentals
The idea
Request correlation
Black-box inference
Schema-based
Metadata propagation
Anatomy of distributed tracing
Sampling
Preserving causality
Inter-request causality
Trace models
Event model
Span model
Clock skew adjustment
Trace analysis
Summary
References
Part II: Data Gathering Problem
Chapter 4: Instrumentation Basics with OpenTracing
Prerequisites
Project source code
Go development environment
Java development environment
Python development environment
MySQL database
Query tools (curl or wget)
Tracing backend (Jaeger)
OpenTracing
Exercise 1 – the Hello application
Hello application in Go
Hello application in Java
Hello application in Python
Exercise summary
Exercise 2 – the first trace
Step 1 – create a tracer instance
Create a tracer in Go
Create a tracer in Java
Create a tracer in Python
Step 2 – start a span
Start a span in Go
Start a span in Java
Start a span in Python
Step 3 – annotate the span
Annotate the span in Go
Annotate the span in Java
Annotate the span in Python
Exercise summary
Exercise 3 – tracing functions and passing context
Step 1 – trace individual functions
Trace individual functions in Go
Trace individual functions in Java
Trace individual functions in Python
Step 2 – combine multiple spans into a single trace
Combine multiple spans into a single trace in Go
Combine multiple spans into a single trace in Java
Combine multiple spans into a single trace in Python
Step 3 – propagate the in-process context
In-process context propagation in Python
In-process context propagation in Java
In-process context propagation in Go
Exercise summary
Exercise 4 – tracing RPC requests
Step 1 – break up the monolith
Microservices in Go
Microservices in Java
Microservices in Python
Step 2 – pass the context between processes
Passing context between processes in Go
Passing context between processes in Java
Passing context between processes in Python
Step 3 – apply OpenTracing-recommended tags
Standard tags in Go
Standard tags in Java
Standard tags in Python
Exercise summary
Exercise 5 – using baggage
Using baggage in Go
Using baggage in Java
Using baggage in Python
Exercise summary
Exercise 6 – auto-instrumentation
Open source instrumentation in Go
Auto-instrumentation in Java
Auto-instrumentation in Python
Exercise 7 – extra credit
Summary
References
Chapter 5: Instrumentation of Asynchronous Applications
Prerequisites
Project source code
Java development environment
Kafka, Zookeeper, Redis, and Jaeger
The Tracing Talk chat application
Implementation
The lib module
The chat-api service
The storage-service microservice
The giphy-service microservice
Running the application
Observing traces
Instrumenting with OpenTracing
Spring instrumentation
Tracer resolver
Redis instrumentation
Kafka instrumentation
Producing messages
Consuming messages
Instrumenting asynchronous code
Summary
References
Chapter 6: Tracing Standards and Ecosystem
Styles of instrumentation
Anatomy of tracing deployment and interoperability
Five shades of tracing
Know your audience
The ecosystem
Tracing systems
Zipkin and OpenZipkin
Jaeger
SkyWalking
X-Ray, Stackdriver, and more
Standards projects
W3C Trace Context
W3C "Data Interchange Format"
OpenCensus
OpenTracing
Summary
References
Chapter 7: Tracing with Service Meshes
Service meshes
Observability via a service mesh
Prerequisites
Project source code
Java development environment
Kubernetes
Istio
The Hello application
Distributed tracing with Istio
Using Istio to generate a service graph
Distributed context and routing
Summary
References
Chapter 8: All About Sampling
Head-based consistent sampling
Probabilistic sampling
Rate limiting sampling
Guaranteed-throughput probabilistic sampling
Adaptive sampling
Local adaptive sampling
Global adaptive sampling
Implications of adaptive sampling
Extensions
Context-sensitive sampling
Ad-hoc or debug sampling
How to deal with oversampling
Post-collection down-sampling
Throttling
Tail-based consistent sampling
Partial sampling
Summary
References
Part III: Getting Value from Tracing
Chapter 9: Turning the Lights On
Tracing as a knowledge base
Service graphs
Deep, path-aware service graphs
Detecting architectural problems
Performance analysis
Critical path analysis
Recognizing trace patterns
Look for error markers
Look for the longest span on the critical path
Look out for missing details
Avoid sequential execution or "staircase"
Be wary when things finish at exactly the same time
Exemplars
Latency histograms
Long-term profiling
Summary
References
Chapter 10: Distributed Context Propagation
Brown Tracing Plane
Pivot tracing
Chaos engineering
Traffic labeling
Testing in production
Debugging in production
Developing in production
Summary
References
Chapter 11: Integration with Metrics and Logs
Three pillars of observability
Prerequisites
Project source code
Java development environment
Running the servers in Docker
Declaring index pattern in Kibana
Running the clients
The Hello application
Integration with metrics
Standard metrics via tracing instrumentation
Adding context to metrics
Context-aware metrics APIs
Integration with logs
Structured logging
Correlating logs with trace context
Context-aware logging APIs
Capturing logs in the tracing system
Do we need separate logging and tracing backends?
Summary
References
Chapter 12: Gathering Insights with Data Mining
Feature extraction
Components of a data mining pipeline
Tracing backend
Trace completion trigger
Feature extractor
Aggregator
Feature extraction exercise
Prerequisites
Project source code
Running the servers in Docker
Defining index mapping in Elasticsearch
Java development environment
Microservices simulator
Running as a Docker image
Running from source
Verify
Define an index pattern in Kibana
The Span Count job
Trace completion trigger
Feature extractor
Observing trends
Beware of extrapolations
Historical analysis
Ad hoc analysis
Summary
References
Part IV: Deploying and Operating Tracing Infrastructure
Chapter 13: Implementing Tracing in Large Organizations
Why is it hard to deploy tracing instrumentation?
Reduce the barrier to adoption
Standard frameworks
In-house adapter libraries
Tracing enabled by default
Monorepos
Integration with existing infrastructure
Where to start
Building the culture
Explaining the value
Integrating with developer workflows
Tracing Quality Metrics
Troubleshooting guide
Don't be on the critical path
Summary
References
Chapter 14: Under the Hood of a Distributed Tracing System
Why host your own?
Customizations and integrations
Bandwidth cost
Own the data
Bet on emerging standards
Architecture and deployment modes
Basic architecture: agent + collector + query service
Client
Agent
Collector
Query service and UI
Data mining jobs
Streaming architecture
Multi-tenancy
Cost accounting
Complete isolation
Granular access controls
Security
Running in multiple DCs
Capturing origin zone
Cross-zone federation
Monitoring and troubleshooting
Resiliency
Over-sampling
Debug traces
Traffic spikes due to DC failover
Perpetual traces
Very long traces
Summary
References
Afterword
References
Other Books You May Enjoy
Index