Learning to build distributed systems is hard, especially if they are large scale. It's not that there is a lack of information out there. You can find academic papers, engineering blogs, and even books on the subject. The problem is that the available information is spread out all over the place, and if you were to put it on a spectrum from theory to practice, you would find a lot of material at the two ends, but not much in the middle.
That is why I decided to write a book to teach the fundamentals of distributed systems so that you don’t have to spend countless hours scratching your head to understand how everything fits together. This is the guide I wished existed when I first started out, and it's based on my experience building large distributed systems that scale to millions of requests per second and billions of devices.
If you develop the back-end of web or mobile applications (or would like to!), this book is for you. When building distributed systems, you need to be familiar with the network stack, data consistency models, scalability and reliability patterns, and much more. Although you can build applications without knowing any of that, you will end up spending hours debugging and re-designing their architecture, learning lessons that you could have acquired in a much faster and less painful way.
Table of contents
1 Introduction
1.1 Communication
1.2 Coordination
1.3 Scalability
1.4 Resiliency
1.5 Operations
1.6 Anatomy of a distributed system
Communication
2 Reliable links
2.1 Reliability
2.2 Connection lifecycle
2.3 Flow control
2.4 Congestion control
2.5 Custom protocols
3 Secure links
3.1 Encryption
3.2 Authentication
3.3 Integrity
3.4 Handshake
4 Discovery
5 APIs
5.1 HTTP
5.2 Resources
5.3 Request methods
5.4 Response status codes
5.5 OpenAPI
5.6 Evolution
Coordination
6 System models
7 Failure detection
8 Time
8.1 Physical clocks
8.2 Logical clocks
8.3 Vector clocks
9 Leader election
9.1 Raft leader election
9.2 Practical considerations
10 Replication
10.1 State machine replication
10.2 Consensus
10.3 Consistency models
10.4 Chain replication
10.5 Solving the CAP theorem
10.6 Coordination avoidance
11 Transactions
11.1 ACID
11.2 Isolation
11.3 Atomicity
11.4 Asynchronous transactions
Scalability
12 Functional decomposition
12.1 Microservices
12.2 API gateway
12.3 CQRS
12.4 Messaging
13 Partitioning
13.1 Sharding strategies
13.2 Rebalancing
14 Duplication
14.1 Network load balancing
14.2 Replication
14.3 Caching
Resiliency
15 Common failure causes
15.1 Single point of failure
15.2 Unreliable network
15.3 Slow processes
15.4 Unexpected load
15.5 Cascading failures
15.6 Risk management
16 Downstream resiliency
16.1 Timeout
16.2 Retry
16.3 Circuit breaker
17 Upstream resiliency
17.1 Load shedding
17.2 Load leveling
17.3 Rate-limiting
17.4 Bulkhead
17.5 Health endpoint
17.6 Watchdog
Testing and operations
18 Testing
18.1 Scope
18.2 Size
18.3 Practical considerations
19 Continuous delivery and deployment
19.1 Review and build
19.2 Pre-production
19.3 Production
19.4 Rollbacks
20 Monitoring
20.1 Metrics
20.2 Service-level indicators
20.3 Service-level objectives
20.4 Alerts
20.5 Dashboards
20.6 On-call
21 Observability
21.1 Logs
21.2 Traces
21.3 Putting it all together
22 Final words
Author(s): Roberto Vitillo
Publisher: Roberto Vitillo
Year: 2021
Language: English
Commentary: Converted copy
Pages: 253
Understanding Distributed Systems
Copyright
About the author
Acknowledgements
Preface
0.1 Who should read this book
1 Introduction
1.1 Communication
1.2 Coordination
1.3 Scalability
1.4 Resiliency
1.5 Operations
1.6 Anatomy of a distributed system
(PART) Communication
Introduction
2 Reliable links
2.1 Reliability
2.2 Connection lifecycle
2.3 Flow control
2.4 Congestion control
2.5 Custom protocols
3 Secure links
3.1 Encryption
3.2 Authentication
3.3 Integrity
3.4 Handshake
4 Discovery
5 APIs
5.1 HTTP
5.2 Resources
5.3 Request methods
5.4 Response status codes
5.5 OpenAPI
5.6 Evolution
(PART) Coordination
Introduction
6 System models
7 Failure detection
8 Time
8.1 Physical clocks
8.2 Logical clocks
8.3 Vector clocks
9 Leader election
9.1 Raft leader election
9.2 Practical considerations
10 Replication
10.1 State machine replication
10.2 Consensus
10.3 Consistency models
10.3.1 Strong consistency
10.3.2 Sequential consistency
10.3.3 Eventual consistency
10.3.4 CAP theorem
10.4 Practical considerations
11 Transactions
11.1 ACID
11.2 Isolation
11.2.1 Concurrency control
11.3 Atomicity
11.3.1 Two-phase commit
11.4 Asynchronous transactions
11.4.1 Log-based transactions
11.4.2 Sagas
11.4.3 Isolation
(PART) Scalability
Introduction
12 Functional decomposition
12.1 Microservices
12.1.1 Benefits
12.1.2 Costs
12.1.3 Practical considerations
12.2 API gateway
12.2.1 Routing
12.2.2 Composition
12.2.3 Translation
12.2.4 Cross-cutting concerns
12.2.5 Caveats
12.3 CQRS
12.4 Messaging
12.4.1 Guarantees
12.4.2 Exactly-once processing
12.4.3 Failures
12.4.4 Backlogs
12.4.5 Fault isolation
12.4.6 Reference plus blob
13 Partitioning
13.1 Sharding strategies
13.1.1 Range partitioning
13.1.2 Hash partitioning
13.2 Rebalancing
13.2.1 Static partitioning
13.2.2 Dynamic partitioning
13.2.3 Practical considerations
14 Duplication
14.1 Network load balancing
14.1.1 DNS load balancing
14.1.2 Transport layer load balancing
14.1.3 Application layer load balancing
14.1.4 Geo load balancing
14.2 Replication
14.2.1 Single leader replication
14.2.2 Multi-leader replication
14.2.3 Leaderless replication
14.3 Caching
14.3.1 Policies
14.3.2 In-process cache
14.3.3 Out-of-process cache
(PART) Resiliency
Introduction
15 Common failure causes
15.1 Single point of failure
15.2 Unreliable network
15.3 Slow processes
15.4 Unexpected load
15.5 Cascading failures
15.6 Risk management
16 Downstream resiliency
16.1 Timeout
16.2 Retry
16.2.1 Exponential backoff
16.2.2 Retry amplification
16.3 Circuit breaker
16.3.1 State machine
17 Upstream resiliency
17.1 Load shedding
17.2 Load leveling
17.3 Rate-limiting
17.3.1 Single-process implementation
17.3.2 Distributed implementation
17.4 Bulkhead
17.5 Health endpoint
17.5.1 Health checks
17.6 Watchdog
(PART) Testing and operations
Introduction
18 Testing
18.1 Scope
18.2 Size
18.3 Practical considerations
19 Continuous delivery and deployment
19.1 Review and build
19.2 Pre-production
19.3 Production
19.4 Rollbacks
20 Monitoring
20.1 Metrics
20.2 Service-level indicators
20.3 Service-level objectives
20.4 Alerts
20.5 Dashboards
20.5.1 Best practices
20.6 On-call
21 Observability
21.1 Logs
21.2 Traces
21.3 Putting it all together
22 Final words
