Kubernetes Best Practices: Blueprints for Building Successful Applications on Kubernetes, 2nd Edition (Final)

Preface
Who Should Read This Book
Why We Wrote This Book
Navigating This Book
New to This Edition
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
1. Setting Up a Basic Service
Application Overview
Managing Configuration Files
Creating a Replicated Service Using Deployments
Best Practices for Image Management
Creating a Replicated Application
Setting Up an External Ingress for HTTP Traffic
Configuring an Application with ConfigMaps
Managing Authentication with Secrets
Deploying a Simple Stateful Database
Creating a TCP Load Balancer by Using Services
Using Ingress to Route Traffic to a Static File Server
Parameterizing Your Application by Using Helm
Deploying Services Best Practices
Summary
2. Developer Workflows
Goals
Building a Development Cluster
Setting Up a Shared Cluster for Multiple Developers
Onboarding Users
Creating and Securing a Namespace
Managing Namespaces
Cluster-Level Services
Enabling Developer Workflows
Initial Setup
Enabling Active Development
Enabling Testing and Debugging
Setting Up a Development Environment Best Practices
Summary
3. Monitoring and Logging in Kubernetes
Metrics Versus Logs
Monitoring Techniques
Monitoring Patterns
Kubernetes Metrics Overview
cAdvisor
Metrics Server
kube-state-metrics
What Metrics Do I Monitor?
Monitoring Tools
Monitoring Kubernetes Using Prometheus
Logging Overview
Tools for Logging
Logging by Using a Loki-Stack
Alerting
Best Practices for Monitoring, Logging, and Alerting
Monitoring
Logging
Alerting
Summary
4. Configuration, Secrets, and RBAC
Configuration Through ConfigMaps and Secrets
ConfigMaps
Secrets
Common Best Practices for the ConfigMap and Secrets APIs
Best Practices Specific to Secrets
RBAC
RBAC Primer
Subjects
Rules
Roles
RoleBindings
RBAC Best Practices
Summary
5. Continuous Integration, Testing, and Deployment
Version Control
Continuous Integration
Testing
Container Builds
Container Image Tagging
Continuous Deployment
Deployment Strategies
Testing in Production
Setting Up a Pipeline and Performing a Chaos Experiment
Setting Up CI
Setting Up CD
Performing a Rolling Upgrade
A Simple Chaos Experiment
Best Practices for CI/CD
Summary
6. Versioning, Releases, and Rollouts
Versioning
Releases
Rollouts
Putting It All Together
Best Practices for Versioning, Releases, and Rollouts
Summary
7. Worldwide Application Distribution and Staging
Distributing Your Image
Parameterizing Your Deployment
Load-Balancing Traffic Around the World
Reliably Rolling Out Software Around the World
Pre-Rollout Validation
Canary Region
Identifying Region Types
Constructing a Global Rollout
When Something Goes Wrong
Worldwide Rollout Best Practices
Summary
8. Resource Management
Kubernetes Scheduler
Predicates
Priorities
Advanced Scheduling Techniques
Pod Affinity and Anti-Affinity
nodeSelector
Taints and Tolerations
Pod Resource Management
Resource Request
Resource Limits and Pod Quality of Service
PodDisruptionBudgets
Minimum available
Maximum unavailable
Managing Resources by Using Namespaces
ResourceQuota
LimitRange
Cluster Scaling
Manual scaling
Cluster autoscaling
Application Scaling
Scaling with HPA
HPA with Custom Metrics
Vertical Pod Autoscaler
Resource Management Best Practices
Summary
9. Networking, Network Security, and Service Mesh
Kubernetes Network Principles
Network Plug-ins
Kubenet
Kubenet Best Practices
The CNI Plug-in
CNI Best Practices
Services in Kubernetes
Service Type ClusterIP
Service Type NodePort
Service Type ExternalName
Service Type LoadBalancer
Ingress and Ingress Controllers
Gateway API
Services and Ingress Controllers Best Practices
Network Security Policy
Network Policy Best Practices
Service Meshes
Service Mesh Best Practices
Summary
10. Pod and Container Security
Pod Security Admission Controller
Enabling Pod Security Admission
Pod Security levels
Activating Pod Security Using Namespace Labels
Workload Isolation and RuntimeClass
Using RuntimeClass
Runtime Implementations
Workload Isolation and RuntimeClass Best Practices
Other Pod and Container Security Considerations
Admission Controllers
Intrusion and Anomaly Detection Tooling
Summary
11. Policy and Governance for Your Cluster
Why Policy and Governance Are Important
How Is This Policy Different?
Cloud Native Policy Engine
Introducing Gatekeeper
Example Policies
Gatekeeper Terminology
Constraint
Rego
Constraint template
Defining Constraint Templates
Defining Constraints
Data Replication
UX
Using Enforcement Action and Audit
Mutation
Testing Policies
Becoming Familiar with Gatekeeper
Policy and Governance Best Practices
Summary
12. Managing Multiple Clusters
Why Multiple Clusters?
Multicluster Design Concerns
Managing Multiple Cluster Deployments
Deployment and Management Patterns
The GitOps Approach to Managing Clusters
Multicluster Management Tools
Kubernetes Federation
Managing Multiple Clusters Best Practices
Summary
13. Integrating External Services with Kubernetes
Importing Services into Kubernetes
Selector-Less Services for Stable IP Addresses
CNAME-Based Services for Stable DNS Names
Active Controller-Based Approaches
Exporting Services from Kubernetes
Exporting Services by Using Internal Load Balancers
Exporting Services on NodePorts
Integrating External Machines and Kubernetes
Sharing Services Between Kubernetes
Third-Party Tools
Connecting Cluster and External Services Best Practices
Summary
14. Running Machine Learning in Kubernetes
Why Is Kubernetes Great for Machine Learning?
Machine Learning Workflow
Machine Learning for Kubernetes Cluster Admins
Model Training on Kubernetes
Training your first model on Kubernetes
Distributed Training on Kubernetes
Resource Constraints
Specialized Hardware
Scheduling idiosyncrasies
Libraries, Drivers, and Kernel Modules
Storage
Dataset storage and distribution among nodes during training
Checkpoints and saving models
Networking
Specialized Protocols
Data Scientist Concerns
Machine Learning on Kubernetes Best Practices
Summary
15. Building Higher-Level Application Patterns on Top of Kubernetes
Approaches to Developing Higher-Level Abstractions
Extending Kubernetes
Extending Kubernetes Clusters
Extending the Kubernetes User Experience
Making Containerized Development Easier
Developing a “Push-to-Deploy” Experience
Design Considerations When Building Platforms
Support Exporting to a Container Image
Support Existing Mechanisms for Service and Service Discovery
Building Application Platforms Best Practices
Summary
16. Managing State and Stateful Applications
Volumes and Volume Mounts
Volume Best Practices
Kubernetes Storage
PersistentVolume
PersistentVolumeClaims
StorageClasses
Container Storage Interface and FlexVolume
Kubernetes Storage Best Practices
Stateful Applications
StatefulSets
Operators
StatefulSet and Operator Best Practices
Summary
17. Admission Control and Authorization
Admission Control
What Are They?
Why Are They Important?
Admission Controller Types
Configuring Admission Webhooks
Admission Control Best Practices
Admission plug-in ordering doesn’t matter
Don’t mutate the same fields
Mutating admission webhooks must be idempotent
Fail open/fail closed
Admission webhooks must respond quickly
Scoping admission webhooks
Always deploy in a separate namespace using NamespaceSelector
Don’t touch the kube-system namespace
Lock down admission webhook configurations with RBAC
Don’t send sensitive data
Authorization
Authorization Modules
ABAC
RBAC
Webhook
Authorization Best Practices
Don’t use ABAC on multiple control plane clusters
Don’t use webhook modules
Summary
18. GitOps and Deployment
What Is GitOps?
Why GitOps?
GitOps Repo Structure
Managing Secrets
Setting Up Flux
GitOps Tooling
GitOps Best Practices
Summary
19. Security
Cluster Security
etcd Access
Authentication
Authorization
TLS
Kubelet and Cloud Metadata Access
Secrets
Logging and Auditing
Cluster Security Posture Tooling
Cluster Security Best Practices
Workload Container Security
Pod Security Admission
Seccomp, AppArmor, and SELinux
Admission Controllers
Operators
Network Policy
Runtime Security
Workload Container Security Best Practices
Code Security
Non-Root and Distroless Containers
Container Vulnerability Scanning
Code Repository Security
Code Security Best Practices
Summary
20. Chaos Testing, Load Testing, and Experiments
Chaos Testing
Goals for Chaos Testing
Prerequisites for Chaos Testing
Chaos Testing Your Application’s Communication
Chaos Testing Your Application’s Operation
Fuzz Testing Your Application for Security and Resiliency
Summary
Load Testing
Goals for Load Testing
Prerequisites for Load Testing
Generating Realistic Traffic
Load Testing Your Application
Tuning Your Application Using Load Tests
Summary
Experiments
Goals for Experiments
Prerequisites for an Experiment
Setting Up an Experiment
Summary
Chaos Testing, Load Testing, and Experiments Summary
21. Implementing an Operator
Operator Key Components
Custom Resource Definitions
Kubernetes API objects, resources, version, group, and kind
Creating Our API
Controller Reconciliation
Resource Validation
Controller Implementation
Operator Life Cycle
Version Upgrades
Operator Best Practices
Summary
22. Conclusion
Index