Deliver lightning fast and reliable messaging for your distributed applications with the flexible and resilient Apache Pulsar platform.
In Apache Pulsar in Action you will learn how to:
• Publish from Apache Pulsar into third-party data repositories and platforms
• Design and develop Apache Pulsar functions
• Perform interactive SQL queries against data stored in Apache Pulsar
Apache Pulsar in Action is a comprehensive and practical guide to building high-traffic applications with Pulsar. You’ll learn to use this mature and battle-tested platform to deliver extreme levels of speed and durability to your messaging. Apache Pulsar committer David Kjerrumgaard teaches you to apply Pulsar’s seamless scalability through hands-on case studies, including IOT analytics applications and a microservices app based on Pulsar functions.
About the technology
Reliable server-to-server messaging is the heart of a distributed application. Apache Pulsar is a flexible real-time messaging platform built to run on Kubernetes and deliver the scalability and resilience required for cloud-based systems. Pulsar supports both streaming and message queuing, and unlike other solutions, it can communicate over multiple protocols including MQTT, AMQP, and Kafka’s binary protocol.
About the book
Apache Pulsar in Action teaches you to build scalable streaming messaging systems using Pulsar. You’ll start with a rapid introduction to enterprise messaging and discover the unique benefits of Pulsar. Following crystal-clear explanations and engaging examples, you’ll use the Pulsar Functions framework to develop a microservices-based application. Real-world case studies illustrate how to implement the most important messaging design patterns.
What's inside
• Publish from Pulsar into third-party data repositories and platforms
• Design and develop Apache Pulsar functions
• Create an event-driven food delivery application
About the reader
Written for experienced Java developers. No prior knowledge of Pulsar required.
About the author
David Kjerrumgaard is a committer on the Apache Pulsar project. He currently serves as a Developer Advocate for StreamNative, where he develops Pulsar best practices and solutions.
Author(s): David Kjerrumgaard
Edition: 1
Publisher: Manning Publications
Year: 2021
Language: English
Commentary: Vector PDF
Pages: 400
City: Shelter Island, NY
Tags: Machine Learning; Neural Networks; Security; Java; Internet of Things; Stream Processing; Messages; Microservices; Apache Kafka; Scalability; High Availability; High Performance; Unit Testing; MQTT; Testing; Data Processing; Data Pipelines; Apache Pulsar
Apache Pulsar in Action
brief contents
contents
foreword
preface
acknowledgments
about this book
Who should read this book
How this book is organized: A roadmap
About the code
Other online resources
liveBook discussion forum
about the author
about the cover illustration
Part 1: Getting started with Apache Pulsar
Chapter 1: Introduction to Apache Pulsar
1.1 Enterprise messaging systems
1.1.1 Key capabilities
1.2 Message consumption patterns
1.2.1 Publish–subscribe messaging
1.2.2 Message queuing
1.3 The evolution of messaging systems
1.3.1 Generic messaging systems
1.3.2 Message-oriented middleware
1.3.3 Enterprise service bus
1.3.4 Distributed messaging systems
1.4 Comparison to Apache Kafka
1.4.1 Multilayered architecture
1.4.2 Message consumption
1.4.3 Data durability
1.4.4 Message acknowledgment
1.4.5 Message retention
1.5 Why do I need Pulsar?
1.5.1 Guaranteed message delivery
1.5.2 Infinite scalability
1.5.3 Resilient to failure
1.5.4 Support for millions of topics
1.5.5 Geo-replication and active failover
1.6 Real-world use cases
1.6.1 Unified messaging systems
1.6.2 Microservices platforms
1.6.3 Connected cars
1.6.4 Fraud detection
Chapter 2: Pulsar concepts and architecture
2.1 Pulsar’s physical architecture
2.1.1 Pulsar’s layered architecture
2.1.2 Stateless serving layer
2.1.3 Stream storage layer
2.1.4 Metadata storage
2.2 Pulsar’s logical architecture
2.2.1 Tenants, namespaces, and topics
2.2.2 Addressing topics in Pulsar
2.2.3 Producers, consumers, and subscriptions
2.2.4 Subscription types
2.3 Message retention and expiration
2.3.1 Data retention
2.3.2 Backlog quotas
2.3.3 Message expiration
2.3.4 Message backlog vs. message expiration
2.4 Tiered storage
Chapter 3: Interacting with Pulsar
3.1 Getting started with Pulsar
3.2 Administering Pulsar
3.2.1 Creating a tenant, namespace, and topic
3.2.2 Java Admin API
3.3 Pulsar clients
3.3.1 The Pulsar Java client
3.3.2 The Pulsar Python client
3.3.3 The Pulsar Go client
3.4 Advanced administration
3.4.1 Persistent topic metrics
3.4.2 Message inspection
Part 2: Apache Pulsar development essentials
Chapter 4: Pulsar functions
4.1 Stream processing
4.1.1 Traditional batching
4.1.2 Micro-batching
4.1.3 Stream native processing
4.2 What is Pulsar Functions?
4.2.1 Programming model
4.3 Developing Pulsar functions
4.3.1 Language native functions
4.3.2 The Pulsar SDK
4.3.3 Stateful functions
4.4 Testing Pulsar functions
4.4.1 Unit testing
4.4.2 Integration testing
4.5 Deploying Pulsar functions
4.5.1 Generating a deployment artifact
4.5.2 Function configuration
4.5.3 Function deployment
4.5.4 The function deployment life cycle
4.5.5 Deployment modes
4.5.6 Pulsar function data flow
Chapter 5: Pulsar IO connectors
5.1 What are Pulsar IO connectors?
5.1.1 Sink connectors
5.1.2 Source connectors
5.1.3 PushSource connectors
5.2 Developing Pulsar IO connectors
5.2.1 Developing a sink connector
5.2.2 Developing a PushSource connector
5.3 Testing Pulsar IO connectors
5.3.1 Unit testing
5.3.2 Integration testing
5.3.3 Packaging Pulsar IO connectors
5.4 Deploying Pulsar IO connectors
5.4.1 Creating and deleting connectors
5.4.2 Debugging deployed connectors
5.5 Pulsar’s built-in connectors
5.5.1 Launching the MongoDB cluster
5.5.2 Link the Pulsar and MongoDB containers
5.5.3 Configure and create the MongoDB sink
5.6 Administering Pulsar IO connectors
5.6.1 Listing connectors
5.6.2 Monitoring connectors
Chapter 6: Pulsar security
6.1 Transport encryption
6.2 Authentication
6.2.1 TLS authentication
6.2.2 JSON Web Token authentication
6.3 Authorization
6.3.1 Roles
6.3.2 An example scenario
6.4 Message encryption
Chapter 7: Schema registry
7.1 Microservice communication
7.1.1 Microservice APIs
7.1.2 The need for a schema registry
7.2 The Pulsar schema registry
7.2.1 Architecture
7.2.2 Schema versioning
7.2.3 Schema compatibility
7.2.4 Schema compatibility check strategies
7.3 Using the schema registry
7.3.1 Modelling the food order event in Avro
7.3.2 Producing food order events
7.3.3 Consuming the food order events
7.3.4 Complete example
7.4 Evolving the schema
Part 3: Hands-on application development with Apache Pulsar
Chapter 8: Pulsar Functions patterns
8.1 Data pipelines
8.1.1 Procedural programming
8.1.2 DataFlow programming
8.2 Message routing patterns
8.2.1 Splitter pattern
8.2.2 Dynamic router pattern
8.2.3 Content-based router pattern
8.3 Message transformation patterns
8.3.1 Message translator pattern
8.3.2 Content enricher pattern
8.3.3 Content filter pattern
Chapter 9: Resiliency patterns
9.1 Pulsar Functions resiliency
9.1.1 Adverse events
9.1.2 Fault detection
9.2 Resiliency design patterns
9.2.1 Retry pattern
9.2.2 Circuit breaker pattern
9.2.3 Rate limiter pattern
9.2.4 Time limiter pattern
9.2.5 Cache pattern
9.2.6 Fallback pattern
9.2.7 Credential refresh pattern
9.3 Multiple layers of resiliency
Chapter 10: Data access
10.1 Data sources
10.2 Data access use cases
10.2.1 Device validation
10.2.2 Driver location data
Chapter 11: Machine learning in Pulsar
11.1 Deploying ML models
11.1.1 Batch processing
11.1.2 Near real-time
11.2 Near real-time model deployment
11.3 Feature vectors
11.3.1 Feature stores
11.3.2 Feature calculation
11.4 Delivery time estimation
11.4.1 ML model export
11.4.2 Feature vector mapping
11.4.3 Model deployment
11.5 Neural nets
11.5.1 Neural net training
11.5.2 Neural net deployment in Java
Chapter 12: Edge analytics
12.1 IIoT architecture
12.1.1 The perception and reaction layer
12.1.2 The transportation layer
12.1.3 The data processing layer
12.2 A Pulsar-based processing layer
12.3 Edge analytics
12.3.1 Telemetric data
12.3.2 Univariate and multivariate
12.4 Univariate analysis
12.4.1 Noise reduction
12.4.2 Statistical analysis
12.4.3 Approximation
12.5 Multivariate analysis
12.5.1 Creating a bidirectional messaging mesh
12.5.2 Multivariate dataset construction
12.6 Beyond the book
appendix A: Running Pulsar on Kubernetes
A.1 Create a Kubernetes cluster
A.1.1 Install prerequisites
A.1.2 Minikube
A.2 The Pulsar Helm chart
A.2.1 What is Helm?
A.2.2 The Pulsar Helm chart
A.3 Using the Pulsar Helm chart
A.3.1 Administering Pulsar on Kubernetes
A.3.2 Configuring clients
appendix B: Geo-replication
B.1 Synchronous geo-replication
B.2 Asynchronous geo-replication
B.2.1 Configuring asynchronous geo-replication
B.3 Asynchronous geo-replication patterns
B.3.1 Multi-active geo-replication
B.3.2 Active-standby geo-replication
B.3.3 Aggregation geo-replication
index
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
Y
Z
