The exponential growth of data combined with the need to derive real-time business value is a critical issue today. An event-driven data mesh can power real-time operational and analytical workloads, all from a single set of data product streams. With practical real-world examples, this book shows you how to successfully design and build an event-driven data mesh.
Building an Event-Driven Data Mesh provides:
• Practical tips for iteratively building your own event-driven data mesh, including hurdles you'll experience, possible solutions, and how to obtain real value as soon as possible
• Solutions to pitfalls you may encounter when moving your organization from monoliths to event-driven architectures
• A clear understanding of how events relate to systems and other events in the same stream and across streams
• A realistic look at event modeling options, such as fact, delta, and command type events, including how these choices will impact your data products
• Best practices for handling events at scale, privacy, and regulatory compliance
• Advice on asynchronous communication and handling eventual consistency
Author(s): Adam Bellemare
Edition: 1
Publisher: O'Reilly Media
Year: 2023
Language: English
Commentary: Publisher's PDF
Pages: 259
City: Sebastopol, CA
Tags: Software Architecture; Event-Driven Architecture; Data Mesh; Data Architecture
Cover
Copyright
Table of Contents
Preface
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Chapter 1. Event-Driven Data Communication
What Is Data Mesh?
An Event-Driven Data Mesh
Using Data in the Operational Plane
The Data Monolith
The Difficulties of Communicating Data for Operational Concerns
The Analytical Plane: Data Warehouses and Data Lakes
The Organizational Impact of Schema on Read
Bad Data: The Costs of Inaction
Can We Unify Analytical and Operational Workflows?
Rethinking Data with Data Mesh
Common Objections to an Event-Driven Data Mesh
Producers Cannot Model Data for Everyone’s Use Cases
Making Multiple Copies of Data Is Bad
Eventual Consistency Is Too Difficult to Manage
Summary
Chapter 2. Data Mesh
Principle 1: Domain Ownership
Domain-Driven Design in Brief
Selecting the Data to Expose from Your Domain
Principle 2: Data as a Product
Data Products Provide Immutable and Time-Stamped Data
Data Products Are Multimodal
Accessing a Data Product Via Push or Pull
The Three Data Product Alignment Types
Event-Driven Data Products as Inputs for Operational Systems
Principle 3: Federated Governance
Specifying Data Product Language, Framework, and API Support
Establishing Data Product Life Cycle Requirements
Establishing Data Handling and Infosec Policies
Identifying and Standardizing Cross-Domain Polysemes
Formalizing Self-Service Platform Requirements
Principle 4: Self-Service Platform
Discovering Data Products and Dependencies
Data Product Management Controls
Data Product Access Controls
Compute and Storage Resources for Building and Using Data Products
Providing Self-Service Through SaaS
Summary
Chapter 3. Event Streams for Data Mesh
Events, Messages, and Records
What’s an Event Stream? What Is It Not?
Ephemeral Message-Passing
Queuing
Consuming and Using Event-Driven Data Products
State Events and Event-Carried State Transfer
Materializing Events
Aggregating Events
The Kappa Architecture
The Lambda Architecture and Why It Doesn’t Work for Data Mesh
Supporting the Requirements for Kappa Architecture
Selecting an Event Broker
Summary
Chapter 4. Federated Governance
Forming a Federated Governance Team
Implementing Standards
Supporting Multimodal Data Product Types
Supporting Data Product Schemas
Supporting Programming Languages and Frameworks
Metadata Standards and Requirements
Ensuring Cross-Domain Data Product Compatibility and Interoperability
Defining and Using Common Entities
Event Stream Keying and Partitioning
Time and Time Zones
What Does a Governance Meeting Look Like?
1. Identifying Existing Problems
2. Drafting Proposals
3. Reviewing Proposals
4. Implementing Proposals
5. Archiving Proposals
Data Security and Access Policies
Disable Data Product Access by Default
Consider End-to-End Encryption
Field-Level Encryption
Data Privacy, the Right to Be Forgotten, and Crypto-Shredding
Data Product Lineage
Topology-Based Lineage
Record-Based Lineage
Summary
Chapter 5. Self-Service Data Platform
The Self-Service Platform Maturity Model
Level 1: The Minimal Viable Platform
The Schema Registry
An Extremely Basic Metadata Catalog
Connectors
Level 1 Wrap-Up: How Does It Work?
Level 2: The Expanded Platform
Full-Featured Metadata Catalog
The Data Product Management Service and UI
Service and User Identities
Basic Access Controls
Stream Processing for Building Data Products
Level 2 Wrap-Up: How Does It Work?
Level 3: The Mature Platform
Authentication, Identification, and Access Management
Integration with Existing Application Delivery Processes
Programmatic Data Product Management API
Monitoring and Alerting
Multiregion and Multicloud Data Products
Level 3 Wrap-Up: How Does It Work?
Summary
Chapter 6. Event Schemas
A Brief Introduction to Serialization and Deserialization
What Is a Schema?
What Are Our Schema Technology Options?
Google’s Protocol Buffers, aka Protobuf
Apache Avro
JSON Schema
Schema Evolution: Changing Your Schemas Through Time
Negotiating a Breaking Schema Change
Step 1: Design the New Data Model
Step 2: Iterate with Your Existing Consumers and the Federated Governance Team
Step 3. Create a Release Schedule, a Data Migration Plan, and a Deprecation Plan
Step 4. Execute the Release
The Role of the Schema Registry
Best Practices for Managing Schemas in Your Codebase
Choosing a Schema Technology
Summary
Chapter 7. Designing Events
Introduction to Event Types
Expanding on State Events and Event-Carried State Transfer
Current State Events
Before/After State Events
Delta Events
Event Sourcing with Delta Events
Why Delta Events Don’t Work for Event-Driven Data Products
Measurement Events
Measurement Events Often Form Aggregate-Aligned Data Products
Measurement Event Sources May Be Lossy
Measurement Events May Power Time-Sensitive Applications
Hybrid Events—State with a Bit of Delta
Notification Events
Summary
Chapter 8. Bootstrapping Data Products
Getting Started: Bootstrapping with Connectors
Dual Writes
Polling the Database to Create Data Products
Change-Data Capture
Change-Data Capture Using a Transactional Outbox
Denormalization and Eventification
Eventification at the Transactional Outbox
Eventification in a Dedicated Service
What Should Go In the Event? And What Should Stay Out?
Slowly Changing Dimensions
Bootstrapping Cloud Storage Files to an Event Stream
Summary
Chapter 9. Integrating Event-Driven Data into Data at Rest
Analytics and the Medallion Architecture
Connecting Event Streams Into Existing Batch-Data Flows
Through the Lens of Data Mesh: What’s Going On?
Through the Lens of Data Mesh: How Do We Solve It?
Balancing File Sizes, SLAs, and Latency
Budget Blues: A Tale of Overspending
Extending the Self-Service Platform for Nonstreaming Data Products
Summary
Chapter 10. Eventual Consistency
Converging on Consistency, One Event at a Time
Strategies for Dealing with Eventual Consistency
Prevent Failures to Avoid Inconsistency
Use Event-Driven Data Products Instead of Request-Response Server API Calls
Expose Eventual Consistency in the Server Response
Plan for New Services and Reprocessing of Data
Synchronize Data Products on Time Boundaries
Out-of-Order Events
Resolving Late-Arriving Events
Summary
Chapter 11. Bringing It All Together
Event Streams for Data Mesh
Integrating with Existing Systems
Operations, Analytics, and Everything in Between
Summary
Index
About the Author
Colophon
