In today's rapidly evolving world of DevOps, traditional silos are a thing of the past. Database administrators are no longer the only experts; site reliability engineers (SREs) and DevOps engineers are database experts as well. This blurring of the lines has led to increased responsibilities, making members of high-performing DevOps teams responsible for end-to-end ownership. This book helps you master DevOps for databases, making it a must-have resource for achieving success in the ever-changing world of DevOps.
You’ll begin by exploring real-world examples of DevOps implementation and its significance in modern data-persistent technologies, before progressing into the various types of database technologies and recognizing their strengths, weaknesses, and commonalities. As you advance, the chapters will teach you about design, implementation, testing, and operations using practical examples, as well as common design patterns, combining them with tooling, technology, and strategies for different types of data-persistent technologies. You’ll also learn how to create complex end-to-end implementation, deployment, and cloud infrastructure strategies defined as code.
By the end of this book, you’ll be equipped with the knowledge and tools to design, build, and operate complex systems efficiently.
Author(s): David Jambor
Edition: 1
Publisher: Packt Publishing Limited
Year: 2023
Language: English
Pages: 446
DevOps for Databases
Contributors
About the author
About the reviewer
Preface
Who this book is for
What this book covers
To get the most out of this book
Conventions used
Get in touch
Share Your Thoughts
Download a free PDF copy of this book
Part 1: Database DevOps
1
Data at Scale with DevOps
The modern data landscape
How do we generate data today?
Why speed matters
Data management strategies
The early days of DevOps
SRE versus DevOps
Engineering principles
Objectives – SLOs/SLIs
Summary
2
Large-Scale Data-Persistent Systems
A brief history of data
The early days of computing
The rise of relational databases
Conclusion
Database evolution
Hierarchical database models
Network database model
Relational databases
OO databases
NoSQL database paradigms
Data warehouses
Architecture
Data modeling
Integration
Data lakes
Architecture
Data ingestion and processing
Storage and retrieval
Security considerations
Conclusion
Summary
3
DBAs in the World of DevOps
The continuously evolving role of the DBA
The rise of data architecture and integration
The emergence of cloud computing and big data
The shift to DevOps
Conclusion
DevOps and DBAs
The role of the database expert in a DevOps-conscious team
Designing and implementing databases
Ensuring high availability and disaster recovery
Performance tuning
Security and compliance
Automation
A proven methodology with quantifiable benefits
Faster time to market
Improved quality
Reduced downtime
Increased scalability
Improved security
Summary
Part 2: Persisting Data in the Cloud
4
Cloud Migration and Modern Data(base) Evolution
What is cloud migration (and why are companies doing it)?
The importance of cloud migration
Steps to keep in mind before cloud migration
Key milestones through cloud migration
Types of cloud migrations
Lift-and-shift migration
Lift-and-reshape migration
Refactoring migration
Hybrid cloud migration
Multi-cloud migration
The process of cloud migration
Monolithic or distributed database systems?
What can a database expert help with during cloud migration?
Data migration strategies and their types
The Big Bang migration strategy
The phased migration strategy
The parallel migration strategy
The hybrid migration strategy
The reverse migration strategy
The ETL strategy
The replication strategy
Why are data migration strategies important during a database cloud migration project?
Taking your migration journey one step further
Summary
5
RDBMS with DevOps
Embracing DevOps
Provisioning and configuration management
Monitoring and alerting
Backup and disaster recovery
Performance optimization
DevSecOps
Summary
6
Non-Relational DMSs with DevOps
Activities and challenges
Data modeling
Denormalization
Nested and dynamic data
Data denormalization
Schema management
Schemaless data modeling
Dynamic schema evolution
Consistency and concurrency control
Deployment automation
Deployment of multiple database engines
Backup and disaster recovery
Capacity planning and scaling
Performance tuning
Data modeling for performance
Distributed query optimization
Network latency and data transfer
Data consistency
Lack of transactions
Eventual consistency
Data sharding
Security
Limited access control
Distributed denial of service attacks
Lack of encryption
Anti-patterns (what not to do…)
Overusing or misusing denormalization
Ignoring or underestimating data consistency
Failing to secure a database
Overlooking performance tuning
Neglecting to plan for growth
Summary
7
AI, ML, and Big Data
Definitions and applications of AI, ML, and big data
The relationship between AI, ML, and big data
The role of DevOps and engineering in AI, ML, and big data
Challenges of AI, ML, and big data
A deep dive into big data as a DevOps data expert
Big data infrastructure
Challenges with big data
A deep dive into ML as a DevOps data expert
How ML works
How ML differs from traditional software applications
Challenges with ML for a DevOps data expert
A deep dive into AI as a DevOps data- expert
Amazon SageMaker
Google Cloud AI platform
Microsoft Azure Machine Learning
Challenges with AI for a DevOps data expert
Summary
Part 3: The Right Tool for the Job
8
Zero-Touch Operations
Traditional versus zero-touch approaches
Automated configuration management
Automated release management
Automated monitoring and alerting
Increased operational efficiency
Automated database provisioning
Automated backup and recovery
Improved reliability and consistency
Automated configuration management
Automated release management
Accelerated deployment and time-to-market
CI/CD pipelines
IaC and orchestration
Enhanced scalability and elasticity
Automated resource provisioning
Container orchestration
Reduced downtime and faster recovery
Automated monitoring and alerting
Streamlined recovery processes
Improved compliance and security
Automated security configurations
Automated compliance checks
Sanity-checking our approach
Conclusion on ROI
Summary
9
Design and Implementation
Designing data-persistence technologies
Database design principles
RDBMS versus NoSQL versus NewSQL
Implementing data-persistence technologies
Installation, configuration, and management of database systems
Practical example – PostgreSQL database server installation, configuration, and management
Disaster recovery planning
Practical example – MongoDB replication and automatic failover
Database provisioning and Infrastructure as Code
Practical example – using Terraform to script the setup of a SQL Server database
Database version control and CI/CD
Importance of database version control
Practical example – using Liquibase to manage database schema changes
Role of the DevOps DBA in CI/CD pipelines
Practical example – Jenkins pipeline with database migrations using Flyway
Database performance tuning
Importance of performance tuning and common strategies
Practical example – optimizing a poorly performing query in Oracle
Security and compliance
Importance of security in database management
Common security measures
Practical example – best practices for securing a MySQL database and ensuring GDPR compliance
Collaboration and communication
Summary
10
Database Automation
Autonomous database management
Self-driving databases – a new horizon in DBMs
Automation of database administration tasks
The implications of self-driving databases
Challenges and future directions
Conclusion
The revolution of performance tuning – from manual to autonomous
Understanding performance tuning
The need for automated performance tuning
Technological foundations of automated performance tuning
The mechanics of automated performance tuning
The implications of automated performance tuning
Challenges and future directions
Conclusion
Automated data lineage tracking – a new era of transparency in data management
Understanding data lineage
The evolution from manual to automated data lineage tracking
The technological foundations of automated data lineage tracking
The process of automated data lineage tracking
The implications of automated data lineage tracking
Challenges and future directions
Conclusion
Data privacy automation – advancing the frontier of privacy compliance in the digital age
Understanding data privacy
The challenges of data privacy
Data masking and anonymization
The advent of data privacy automation
Technological underpinnings of data privacy automation
The process of data privacy automation
The benefits and implications of data privacy automation
Conclusion
Automated data discovery and cataloging – unveiling the hidden treasures in today’s data landscape
Understanding data discovery and cataloging
The growing need for automation in data discovery and cataloging
What is automated data discovery and cataloging?
The key features of automated data discovery and cataloging tools
The process of automated data discovery and cataloging
The benefits and implications of automated data discovery and cataloging
Conclusion
The ascendancy of DBaaS – transforming business efficiency and data utilization in the digital age
Understanding DBaaS
The mechanics of DBaaS
The impact of DBaaS on business operations
DBaaS – the future of database management
Conclusion
The emergence of serverless databases – revolutionizing DBaaS through on-demand scalability and cost efficiency
Understanding serverless databases
Why serverless databases? The driving forces
How serverless databases work
The impact of serverless databases on business operations
The future of serverless databases
Conclusion
Summary
Part 4: Build and Operate
11
End-to-End Ownership Model – a Theoretical Case Study
End-to-end ownership – a case study
Adoption of the end-to-end ownership model
Setting the stage
Project initiation
Formation of cross-functional teams
Defining end-to-end ownership
Design and development phase
Collaborative design and planning
Agile development practices
CI and continuous testing
Deployment and release
IaC
CD pipelines
Canary releases and feature flags
Monitoring and IM
Proactive monitoring and alerting
IR and post-mortems
Continuous improvement
Feedback and iteration
Gathering user feedback
Prioritizing and implementing changes
A/B testing and experiments
Scaling and challenges
Scaling the end-to-end ownership model
Managing dependencies
Balancing autonomy and alignment
Summary
12
Immutable and Idempotent Logic – A Theoretical Case Study
Introduction to immutable and idempotent logic
Immutable logic in data-persisting technologies
Understanding immutability in the context of data storage
Benefits and use cases of immutable data storage
Examples of immutable data storage approaches
Implementing immutable logic with data-persisting technologies
Idempotent logic in data-persisting technologies
Introduction to idempotent operations and their significance
Examples of idempotent operations in data persistence
Ensuring idempotency in data processing and transformations
Practical examples and use cases
Immutable and idempotent logic in relational databases
Immutable and idempotent approaches in NoSQL databases
Immutable and idempotent patterns in distributed storage systems
Considerations and best practices
Performance and scalability implications of immutable and idempotent approaches
Data consistency and integrity considerations
Handling failures and retries with idempotent logic
Managing data evolution and schema changes with immutability
Future trends and challenges
Emerging technologies and advancements in data persistence
Integrating immutable and idempotent logic with cloud-native architectures
Addressing complexities and trade-offs in large-scale data persistence systems
Summary
13
Operators and Self-Healing Data Persistent Systems
Self-healing systems
Components of a self-healing system
Importance of self-healing systems
Risks and limitations
Technical example of each core principle of a self-healing system
Operators in Kubernetes
Overview of Kubernetes and containerization
Understanding operators
Operator frameworks and ecosystem
Benefits of operators in Kubernetes
Use cases of operators in Kubernetes
Self-healing databases
Traditional database challenges
Self-healing mechanisms in databases
Benefits of self-healing databases
Risks and limitations
Factors influencing self-healing in different databases
Relational databases
NoSQL databases
NewSQL databases
Time-series databases
Self-healing in Kubernetes – implementation and best practices
Key components for self-healing in Kubernetes
Implementing self-healing in Kubernetes – best practices
Challenges and considerations
Conclusion
Case studies – self-healing databases in Kubernetes
Case study 1 – MySQL Operator
Case study 2 – MongoDB Operator
Case study 3 – Cassandra Operator
Benefits of self-healing databases in Kubernetes
Challenges and future directions
Challenges in self-healing systems
Future directions
Summary
14
Bringing Them Together
Alex’s AI journey
Introduction and project assignment
Software and infrastructure architecture decisions
Relational versus non-relational databases
Implementing caching, data lakes, and data warehouses
Security concerns and solutions
First status update
Implementation
Utilizing DevOps and SRE methodologies
The power of immutable and idempotent logic
Embracing zero-touch automation
Update 2
Implementing self-healing systems
Implementing load balancers and scaling
Update 3
Observability and operations
The art of canary deployments
Database scaling
Security and compliance in operations
Update 4
Version-controlled environment variables
Update 5
Lessons learned and future directions
Summary
Part 5: The Future of Data
15
Specializing in Data
Mastering data – bridging the gap between IT and business
Data-driven decision-making for business executives
Building a data-driven culture – an enterprise perspective
My first experience, Unix – 2009
The first signs of DevOps – 2010s
Support and software engineering in 2012
My first SRE team – 2015
Steep learning curves – 2017
Putting it all into practice – 2019
The landscape of 2023 – data and DevOps united
Integration of DevOps with data engineering
DataOps – revolutionizing data analytics pipelines
MLOps – bridging the gap between ML development and operations
AI-powered DevOps/SRE
Application of SRE principles to data systems
DevSecOps – security in the age of data
Summary
16
The Exciting New World of Data
Part 1 – the future of data-persisting technologies
The evolution of current data-persisting technologies
Emerging data-persisting technologies
Future use cases and challenges
Part 2 – anticipated changes in AI/ML DevOps
Advancements in MLOps
Future use cases and challenges
Career impacts and future skill requirements
Part 3 – evolving trends in SRE
Changes in the SRE landscape
The role of SRE in future IT operations
Part 4 – career outlook and emerging skill sets in SRE
SRE careers – opportunities, skills, and preparation
Career opportunities in SRE
Skill set requirements
Preparing for a career in SRE
Innovations in data persistence technologies
The current state of data persistence technologies
Future outlook – next-generation data persistence technologies
Preparing for the future – skills and strategies in data persistence technologies
Part 5 – the future of designing, building, and operating cutting-edge systems
Emerging technologies in system design and development
Potential implications and challenges in designing, building, and operating cutting-edge systems
Strategies for success – preparing for the future of designing, building, and operating cutting-edge systems
Summary
Index
Why subscribe?
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts
Download a free PDF copy of this book
