Python for DevOps: Learn Ruthlessly Effective Automation

Cover
Copyright
Table of Contents
Preface
What Does DevOps Mean to the Authors?
How to Use This Book
Conceptual Topics
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Noah
Kennedy
Alfredo
Grig
Chapter 1. Python Essentials for DevOps
Installing and Running Python
The Python Shell
Jupyter Notebooks
Procedural Programming
Variables
Basic Math
Comments
Built-in Functions
Print
Range
Execution Control
if/elif/else
for Loops
while Loops
Handling Exceptions
Built-in Objects
What Is an Object?
Object Methods and Attributes
Sequences
Functions
Anatomy of a Function
Functions as Objects
Anonymous Functions
Using Regular Expressions
Searching
Character Sets
Character Classes
Groups
Named Groups
Find All
Find Iterator
Substitution
Compiling
Lazy Evaluation
Generators
Generator Comprehensions
More IPython Features
Using IPython to Run Unix Shell Commands
Exercises
Chapter 2. Automating Files and the Filesystem
Reading and Writing Files
Using Regular Expressions to Search Text
Dealing with Large Files
Encrypting Text
Hashing with Hashlib
Encryption with Cryptography
The os Module
Managing Files and Directories Using os.path
Walking Directory Trees Using os.walk
Paths as Objects with Pathlib
Chapter 3. Working with the Command Line
Working with the Shell
Talking to the Interpreter with the sys Module
Dealing with the Operating System Using the os Module
Spawn Processes with the subprocess Module
Creating Command-Line Tools
Using sys.argv
Using argparse
Using click
fire
Implementing Plug-ins
Case Study: Turbocharging Python with Command-Line Tools
Using the Numba Just-in-Time (JIT) Compiler
Using the GPU with CUDA Python
Running True Multicore Multithreaded Python Using Numba
KMeans Clustering
Exercises
Chapter 4. Useful Linux Utilities
Disk Utilities
Measuring Performance
Partitions
Retrieving Specific Device Information
Network Utilities
SSH Tunneling
Benchmarking HTTP with Apache Benchmark (ab)
Load Testing with molotov
CPU Utilities
Viewing Processes with htop
Working with Bash and ZSH
Customizing the Python Shell
Recursive Globbing
Searching and Replacing with Confirmation Prompts
Removing Temporary Python Files
Listing and Filtering Processes
Unix Timestamp
Mixing Python with Bash and ZSH
Random Password Generator
Does My Module Exist?
Changing Directories to a Module’s Path
Converting a CSV File to JSON
Python One-Liners
Debuggers
How Fast Is this Snippet?
strace
Exercises
Case Study Question
Chapter 5. Package Management
Why Is Packaging Important?
When Packaging Might Not Be Needed
Packaging Guidelines
Descriptive Versioning
The changelog
Choosing a Strategy
Packaging Solutions
Native Python Packaging
Debian Packaging
RPM Packaging
Management with systemd
Long-Running Processes
Setting It Up
The systemd Unit File
Installing the Unit
Log Handling
Exercises
Case Study Question
Chapter 6. Continuous Integration and Continuous Deployment
Real-World Case Study: Converting a Poorly Maintained WordPress Site to Hugo
Setting Up Hugo
Converting WordPress to Hugo Posts
Creating an Algolia Index and Updating It
Orchestrating with a Makefile
Deploying with AWS CodePipeline
Real-World Case Study: Deploying a Python App Engine Application with Google Cloud Build
Real-World Case Study: NFSOPS
Chapter 7. Monitoring and Logging
Key Concepts in Building Reliable Systems
Immutable DevOps Principles
Centralized Logging
Case Study: Production Database Kills Hard Drives
Did You Build It or Buy It?
Fault Tolerance
Monitoring
Graphite
StatsD
Prometheus
Instrumentation
Naming Conventions
Logging
Why Is It Hard?
The basicconfig
Deeper Configuration
Common Patterns
The ELK Stack
Logstash
Elasticsearch and Kibana
Exercises
Case Study Question
Chapter 8. Pytest for DevOps
Testing Superpowers with pytest
Getting Started with pytest
Testing with pytest
Differences with unittest
pytest Features
conftest.py
The Amazing assert
Parametrization
Fixtures
Getting Started
Built-in Fixtures
Infrastructure Testing
What Is System Validation?
Introduction to Testinfra
Connecting to Remote Nodes
Features and Special Fixtures
Examples
Testing Jupyter Notebooks with pytest
Exercises
Case Study Question
Chapter 9. Cloud Computing
Cloud Computing Foundations
Types of Cloud Computing
Types of Cloud Services
Infrastructure as a Service
Metal as a Service
Platform as a Service
Serverless Computing
Software as a Service
Infrastructure as Code
Continuous Delivery
Virtualization and Containers
Hardware Virtualization
Software Defined Networks
Software Defined Storage
Containers
Challenges and Opportunities in Distributed Computing
Python Concurrency, Performance, and Process Management in the Cloud Era
Process Management
Manage Processes with Subprocess
Using Multiprocessing to Solve Problems
Forking Processes with Pool()
Function as a Service and Serverless
High Performance Python with Numba
Using Numba Just in Time Compiler
Using High-Performance Servers
Conclusion
Exercises
Case Study Questions
Chapter 10. Infrastructure as Code
A Classification of Infrastructure Automation Tools
Manual Provisioning
Automated Infrastructure Provisioning with Terraform
Provisioning an S3 Bucket
Provisioning an SSL Certificate with AWS ACM
Provisioning an Amazon CloudFront Distribution
Provisioning a Route 53 DNS Record
Copying Static Files to S3
Deleting All AWS Resources Provisioned with Terraform
Automated Infrastructure Provisioning with Pulumi
Creating a New Pulumi Python Project for AWS
Creating Configuration Values for the Staging Stack
Provisioning an ACM SSL Certificate
Provisioning a Route 53 Zone and DNS Records
Provisioning a CloudFront Distribution
Provisioning a Route 53 DNS Record for the Site URL
Creating and Deploying a New Stack
Exercises
Chapter 11. Container Technologies: Docker and Docker Compose
What Is a Docker Container?
Creating, Building, Running, and Removing Docker Images and Containers
Publishing Docker Images to a Docker Registry
Running a Docker Container with the Same Image on a Different Host
Running Multiple Docker Containers with Docker Compose
Porting the docker-compose Services to a New Host and Operating System
Exercises
Chapter 12. Container Orchestration: Kubernetes
Short Overview of Kubernetes Concepts
Using Kompose to Create Kubernetes Manifests from docker-compose.yaml
Deploying Kubernetes Manifests to a Local Kubernetes Cluster Based on minikube
Launching a GKE Kubernetes Cluster in GCP with Pulumi
Deploying the Flask Example Application to GKE
Installing Prometheus and Grafana Helm Charts
Destroying the GKE Cluster
Exercises
Chapter 13. Serverless Technologies
Deploying the Same Python Function to the “Big Three” Cloud Providers
Installing Serverless Framework
Deploying Python Function to AWS Lambda
Deploying Python Function to Google Cloud Functions
Deploying Python Function to Azure
Deploying a Python Function to Self-Hosted FaaS Platforms
Deploying Python Function to OpenFaaS
Provisioning DynamoDB Table, Lambda Functions, and API Gateway Methods Using the AWS CDK
Exercises
Chapter 14. MLOps and Machine learning Engineering
What Is Machine Learning?
Supervised Machine Learning
Modeling
Python Machine learning Ecosystem
Deep Learning with PyTorch
Cloud Machine learning Platforms
Machine learning Maturity Model
Machine Learning Key Terminology
Level 1: Framing, Scope Identification, and Problem Definition
Level 2: Continuous Delivery of Data
Level 3: Continuous Delivery of Clean Data
Level 4: Continuous Delivery of Exploratory Data Analysis
Level 5: Continuous Delivery of Traditional ML and AutoML
Level 6: ML Operational Feedback Loop
Sklearn Flask with Kubernetes and Docker
Sklearn Flask with Kubernetes and Docker
EDA
Modeling
Tune Scaled GBM
Fit Model
Evaluate
adhoc_predict
JSON Workflow
Scale Input
adhoc_predict from Pickle
Scale Input
Exercises
Case Study Question
Learning Assessments
Chapter 15. Data Engineering
Small Data
Dealing with Small Data Files
Write a File
Read a File
Generator Pipeline to Read and Process Lines
Using YAML
Big Data
Big Data Tools, Components, and Platforms
Data Sources
Filesystems
Data Storage
Real-Time Streaming Ingestion
Case Study: Building a Homegrown Data Pipeline
Serverless Data Engineering
Using AWS Lambda with CloudWatch Events
Using Amazon CloudWatch Logging with AWS Lambda
Using AWS Lambda to Populate Amazon Simple Queue Service
Wiring Up CloudWatch Event Trigger
Creating Event-Driven Lambdas
Reading Amazon SQS Events from AWS Lambda
Conclusion
Exercises
Case Study Question
Chapter 16. DevOps War Stories and Interviews
Film Studio Can’t Make Film
Game Studio Can’t Ship Game
Python Scripts Take 60 Seconds to Launch
Putting Out a Fire with a Cache and Intelligent Instrumentation
You’ll Automate Yourself Out of a Job!
DevOps Antipatterns
No Automated Build Server Antipattern
Flying Blind
Difficulties in Coordination as an Ongoing Accomplishment
No Teamwork
Interviews
Glenn Solomon
Andrew Nguyen
Gabriella Roman
Rigoberto Roche
Jonathan LaCour
Ville Tuulos
Joseph Reis
Teijo Holzer
Matt Harrison
Michael Foord
Recommendations
Exercises
Challenges
Capstone Project
Index
About the Authors
Colophon