Kafka The Definitive Guide

Cover
Confluent
Copyright
Table of Contents
Foreword to the Second Edition
Foreword to the First Edition
Preface
Who Should Read This Book
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Chapter 1. Meet Kafka
Publish/Subscribe Messaging
How It Starts
Individual Queue Systems
Enter Kafka
Messages and Batches
Schemas
Topics and Partitions
Producers and Consumers
Brokers and Clusters
Multiple Clusters
Why Kafka?
Multiple Producers
Multiple Consumers
Disk-Based Retention
Scalable
High Performance
Platform Features
The Data Ecosystem
Use Cases
Kafka’s Origin
LinkedIn’s Problem
The Birth of Kafka
Open Source
Commercial Engagement
The Name
Getting Started with Kafka
Chapter 2. Installing Kafka
Environment Setup
Choosing an Operating System
Installing Java
Installing ZooKeeper
Installing a Kafka Broker
Configuring the Broker
General Broker Parameters
Topic Defaults
Selecting Hardware
Disk Throughput
Disk Capacity
Memory
Networking
CPU
Kafka in the Cloud
Microsoft Azure
Amazon Web Services
Configuring Kafka Clusters
How Many Brokers?
Broker Configuration
OS Tuning
Production Concerns
Garbage Collector Options
Datacenter Layout
Colocating Applications on ZooKeeper
Summary
Chapter 3. Kafka Producers: Writing Messages to Kafka
Producer Overview
Constructing a Kafka Producer
Sending a Message to Kafka
Sending a Message Synchronously
Sending a Message Asynchronously
Configuring Producers
client.id
acks
Message Delivery Time
linger.ms
buffer.memory
compression.type
batch.size
max.in.flight.requests.per.connection
max.request.size
receive.buffer.bytes and send.buffer.bytes
enable.idempotence
Serializers
Custom Serializers
Serializing Using Apache Avro
Using Avro Records with Kafka
Partitions
Headers
Interceptors
Quotas and Throttling
Summary
Chapter 4. Kafka Consumers: Reading Data from Kafka
Kafka Consumer Concepts
Consumers and Consumer Groups
Consumer Groups and Partition Rebalance
Static Group Membership
Creating a Kafka Consumer
Subscribing to Topics
The Poll Loop
Thread Safety
Configuring Consumers
fetch.min.bytes
fetch.max.wait.ms
fetch.max.bytes
max.poll.records
max.partition.fetch.bytes
session.timeout.ms and heartbeat.interval.ms
max.poll.interval.ms
default.api.timeout.ms
request.timeout.ms
auto.offset.reset
enable.auto.commit
partition.assignment.strategy
client.id
client.rack
group.instance.id
receive.buffer.bytes and send.buffer.bytes
offsets.retention.minutes
Commits and Offsets
Automatic Commit
Commit Current Offset
Asynchronous Commit
Combining Synchronous and Asynchronous Commits
Committing a Specified Offset
Rebalance Listeners
Consuming Records with Specific Offsets
But How Do We Exit?
Deserializers
Custom Deserializers
Using Avro Deserialization with Kafka Consumer
Standalone Consumer: Why and How to Use a Consumer Without a Group
Summary
Chapter 5. Managing Apache Kafka Programmatically
AdminClient Overview
Asynchronous and Eventually Consistent API
Options
Flat Hierarchy
Additional Notes
AdminClient Lifecycle: Creating, Configuring, and Closing
client.dns.lookup
request.timeout.ms
Essential Topic Management
Configuration Management
Consumer Group Management
Exploring Consumer Groups
Modifying Consumer Groups
Cluster Metadata
Advanced Admin Operations
Adding Partitions to a Topic
Deleting Records from a Topic
Leader Election
Reassigning Replicas
Testing
Summary
Chapter 6. Kafka Internals
Cluster Membership
The Controller
KRaft: Kafka’s New Raft-Based Controller
Replication
Request Processing
Produce Requests
Fetch Requests
Other Requests
Physical Storage
Tiered Storage
Partition Allocation
File Management
File Format
Indexes
Compaction
How Compaction Works
Deleted Events
When Are Topics Compacted?
Summary
Chapter 7. Reliable Data Delivery
Reliability Guarantees
Replication
Broker Configuration
Replication Factor
Unclean Leader Election
Minimum In-Sync Replicas
Keeping Replicas In Sync
Persisting to Disk
Using Producers in a Reliable System
Send Acknowledgments
Configuring Producer Retries
Additional Error Handling
Using Consumers in a Reliable System
Important Consumer Configuration Properties for Reliable Processing
Explicitly Committing Offsets in Consumers
Validating System Reliability
Validating Configuration
Validating Applications
Monitoring Reliability in Production
Summary
Chapter 8. Exactly-Once Semantics
Idempotent Producer
How Does the Idempotent Producer Work?
Limitations of the Idempotent Producer
How Do I Use the Kafka Idempotent Producer?
Transactions
Transactions Use Cases
What Problems Do Transactions Solve?
How Do Transactions Guarantee Exactly-Once?
What Problems Aren’t Solved by Transactions?
How Do I Use Transactions?
Transactional IDs and Fencing
How Transactions Work
Performance of Transactions
Summary
Chapter 9. Building Data Pipelines
Considerations When Building Data Pipelines
Timeliness
Reliability
High and Varying Throughput
Data Formats
Transformations
Security
Failure Handling
Coupling and Agility
When to Use Kafka Connect Versus Producer and Consumer
Kafka Connect
Running Kafka Connect
Connector Example: File Source and File Sink
Connector Example: MySQL to Elasticsearch
Single Message Transformations
A Deeper Look at Kafka Connect
Alternatives to Kafka Connect
Ingest Frameworks for Other Datastores
GUI-Based ETL Tools
Stream Processing Frameworks
Summary
Chapter 10. Cross-Cluster Data Mirroring
Use Cases of Cross-Cluster Mirroring
Multicluster Architectures
Some Realities of Cross-Datacenter Communication
Hub-and-Spoke Architecture
Active-Active Architecture
Active-Standby Architecture
Stretch Clusters
Apache Kafka’s MirrorMaker
Configuring MirrorMaker
Multicluster Replication Topology
Securing MirrorMaker
Deploying MirrorMaker in Production
Tuning MirrorMaker
Other Cross-Cluster Mirroring Solutions
Uber uReplicator
LinkedIn Brooklin
Confluent Cross-Datacenter Mirroring Solutions
Summary
Chapter 11. Securing Kafka
Locking Down Kafka
Security Protocols
Authentication
SSL
SASL
Reauthentication
Security Updates Without Downtime
Encryption
End-to-End Encryption
Authorization
AclAuthorizer
Customizing Authorization
Security Considerations
Auditing
Securing ZooKeeper
SASL
SSL
Authorization
Securing the Platform
Password Protection
Summary
Chapter 12. Administering Kafka
Topic Operations
Creating a New Topic
Listing All Topics in a Cluster
Describing Topic Details
Adding Partitions
Reducing Partitions
Deleting a Topic
Consumer Groups
List and Describe Groups
Delete Group
Offset Management
Dynamic Configuration Changes
Overriding Topic Configuration Defaults
Overriding Client and User Configuration Defaults
Overriding Broker Configuration Defaults
Describing Configuration Overrides
Removing Configuration Overrides
Producing and Consuming
Console Producer
Console Consumer
Partition Management
Preferred Replica Election
Changing a Partition’s Replicas
Dumping Log Segments
Replica Verification
Other Tools
Unsafe Operations
Moving the Cluster Controller
Removing Topics to Be Deleted
Deleting Topics Manually
Summary
Chapter 13. Monitoring Kafka
Metric Basics
Where Are the Metrics?
What Metrics Do I Need?
Application Health Checks
Service-Level Objectives
Service-Level Definitions
What Metrics Make Good SLIs?
Using SLOs in Alerting
Kafka Broker Metrics
Diagnosing Cluster Problems
The Art of Under-Replicated Partitions
Broker Metrics
Topic and Partition Metrics
JVM Monitoring
OS Monitoring
Logging
Client Monitoring
Producer Metrics
Consumer Metrics
Quotas
Lag Monitoring
End-to-End Monitoring
Summary
Chapter 14. Stream Processing
What Is Stream Processing?
Stream Processing Concepts
Topology
Time
State
Stream-Table Duality
Time Windows
Processing Guarantees
Stream Processing Design Patterns
Single-Event Processing
Processing with Local State
Multiphase Processing/Repartitioning
Processing with External Lookup: Stream-Table Join
Table-Table Join
Streaming Join
Out-of-Sequence Events
Reprocessing
Interactive Queries
Kafka Streams by Example
Word Count
Stock Market Statistics
ClickStream Enrichment
Kafka Streams: Architecture Overview
Building a Topology
Optimizing a Topology
Testing a Topology
Scaling a Topology
Surviving Failures
Stream Processing Use Cases
How to Choose a Stream Processing Framework
Summary
Appendix A. Installing Kafka on Other Operating Systems
Installing on Windows
Using Windows Subsystem for Linux
Using Native Java
Installing on macOS
Using Homebrew
Installing Manually
Appendix B. Additional Kafka Tools
Comprehensive Platforms
Cluster Deployment and Management
Monitoring and Data Exploration
Client Libraries
Stream Processing
Index
About the Authors
Colophon