Linux System Administration For The 2020s: The Modern Sysadmin Leaving Behind The Culture Of Build And Maintain

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Part I: Laying the Foundation
Chapter 1: Linux at a Glance
Brief Unix to Linux History
Open Source
Linux Is Everywhere
Community Linux Distributions
Community
Upstream
Community Contributors
Common Distributions
Which Distribution Is Best for You
Before Committing
The Three Linux Distro Categories
Option One: Out-of-the-Box Distros
Easy to Understand
Installation Should Not Require a Degree
Try Ubuntu
Walk Before Running
Option Two: The Almost Out-of-the-Box Distros
Try Fedora, openSUSE, or Debian
Option Three: The “Challenge Accepted” Distros
With Great Power …
Try Arch Linux or Gentoo
Enterprise Linux Distributions
Red Hat
Red Hat Enterprise Linux
Automation
Hybrid Cloud
Canonical
Linux Support
Cloud
Internet of Things
SUSE
Server and Desktop
Cloud, Storage, and Management
Community vs. Enterprise
Knowledge Check
Summary
Part II: Strengthening Core Skills
Chapter 2: New Tools to Improve the Administrative Experience
Task Management
Starting a Process
Task Visualization Tooling
Top
Alternatives to Top
nmon
Killing Processes
Zombie Processes
Background Tasks
Running Time-Consuming Tasks
Screen
Tmux
Ansible Introduction
Installing Ansible
Package Management
Pip
Configuring Ansible
Ansible Inventory
Running Ansible
Playbooks
Roles
Role Directory Structure
Generating Ansible Roles
Modules
Sharing Your Ansible
Ansible Galaxy
Web Consoles
Cockpit
Installation
Configuration
Using Cockpit
Limitations
Alternatives to Cockpit
Webmin
Ajenti
Text Consoles
Installing
Using
Summary
Chapter 3: Estate Management
Outdated Ways of Working
Outdated Skills
Keeping Knowledge to Themselves
Over Engineering
Shell Scripting
Snowflakes
Reinventing the Wheel
Build Process
Manual Installation Methods
Boot Media Install
Network Install
Templates
Virtual Machine Images
Automated Linux Installations
Method 1: Network Install
PXE Server
Kickstart
Method 2: Virtual Machine Templates
Hypervisor API
Ansible Examples
Using Images
Golden Image
Use It
Don’t Use It
Image Catalog
Advantages
Disadvantages
Build Process Flow
Basic Build Process
What Can Be Improved
Automate, Automate, Automate
Introduce a User Request Portal
Integration with Other Platforms
Simplify Resource Requirements
Use an Automation Platform
Introduce Expiry Dates
Automated Build Process Flow
System Patching
Update Types
Package Updates
Errata
Staging
Patch Management Systems
Planning
Rollback
System Restore from Backup
Restore Snapshot
Package Management Rollback
Reinstallation of Packages
Redeployment of System
Backup and Recovery
Important Directories and Files
Virtual Machine Backups
Disaster Recovery
Best Strategies Based on Recovery Times
Replicated Data Centers
Stretched Clusters
Infrastructure As Code
Cloud
Common Bad Practices
Virtual Machine Templates
Patching or Lack Thereof
Firewall Disabled
SELinux Disabled or Permissive
Using Community Repositories
Scripts, Scripts, and More Scripts
Running As Root
Good Practices
Building Throwaway Systems
Automate As Much As Possible
Search Before Creating
Sharing Knowledge and Collaborating
Source Control
Reassessing System Requirements
Summary
Chapter 4: Estate Management Tools
Management Systems
Linux Platform Tools
Linux Platform Tools Available
Selecting Your Linux Platform Tool
The Decision
Satellite Server
Satellite 5
Configuration Management
System Deployment
Satellite 6
Content Management
Content Views
Life Cycles
Content Management Flow
System Provisioning
System Patching
Configuration Management
Reasons to Use Satellite
Reasons to Not Use Satellite
SUSE Manager
Uyuni
Support
SUSE Manager Configuration
Reasons to Use SUSE Manager
Reasons to Not Use SUSE Manager
Foreman
Provision Hypervisors
Plugins
Open Source Does Need Money Too
Spacewalk
Abandoned
Why It Was Good
Network Provisioning
Environment Staging
Thank You for Your Service
Provisioning Tools
Cloudforms
Single Pane of Glass
State Machines
User Request Portal
Chargeback
Request Approvals
Advantages
Disadvantages
Terraform
Products Available
Community CLI
Terraform Cloud Platform
API and Extracting Useful Information
Don’t Reinvent the Wheel
Why to Not Write Your Own Tool
Best Tools to Use
Pipeline Tooling
Automation Platforms
Shell Scripts
Summary
Chapter 5: Automation
Automation in Theory
Idempotent Code
Knowing When and When Not to Automate
Reasons to Automate
Reasons Not to Automate
State Management
Automation Tooling
Automation Scripting Languages
YAML
These Are Not the Spaces You Are Looking For
YAML in Action
Ansible
SaltStack
Ruby
Python
Shell Scripting
Automation Platforms
Automation in Estate Management Tools
Reasons to Use
Reasons Not to Use
Ansible Automation Platform
Agentless
Potential Security Hole
Using Ansible
Command Line
Graphical User Interface
Reasons to Use Ansible
Reasons Not to Use Ansible
AWX
Reasons to Use AWX
Reasons Not to Use AWX
SaltStack
Server to Client Communication
Remote Execution
Configuration Management
Uses a Message Bus
Reasons to Use SaltStack
Reasons to Not Use SaltStack
Puppet
Red Hat and Puppet
Server Agent Based
Potential Lower Adoption
Enterprise and Community
Reasons to Use Puppet
Reasons to Not Use Puppet
Chef
Ways to Use Chef
Managed Service
On-Premise
Community
Reasons to Use Chef
Reasons to Not Use Chef
Making the Decision
Market Trends
See for Yourself
Enterprise vs. Community vs. Cost
Product Life Cycle
Automation with Management Tools
State Management
Enterprise Products
Use Case Example
The Platform Tool
The Platform Tool Configuration
The Mistake
Laying in the Shadows Waiting
Safety Net
Setting Up a SOE
Build from a Standard
Source Control
Phased Testing
Code Development
Code Testing and Peer Reviewed
Code Promotion
Automate the Automation
Self-Healing
Self-Healing Layers
Removing All Single Points of Failure
Hardware Layer Self-Healing
Reporting
Ensuring Platform Availability
Automated Recovery
Platform Layer Self-Healing
Application Layer Self-Healing
When to Self-Heal
How to Implement Self-Healing
Gates
Tooling: Automation and State Management
Machine Learning
Off-the-Shelf Products
Dynatrace
Automation Best Practices
Do Not Reinvent the Wheel, Again …
Code Libraries
Ansible
Puppet
SaltStack
Metadata
Things to Avoid
Shell Scripts
Restarting Services When Not Required
Using Old Versions
Correct Version Documentation
Good Practices
Debugging
Don't Forget README
Source Control
Summary
Chapter 6: Containers
Getting Started
Virtual Machine vs. Container
Container History
Container Runtimes
Low-Level or OCI Runtimes
Native Runtimes
Virtual and Sandboxed Runtimes
Sandbox Runtimes
Virtual Runtimes
Container Runtime Interface
Containerd
CRI-O
Container Engines
Docker
Podman
Container Images
Container Registries
Cloud Registries
Local Registries
Container Registry Providers
Containers in Practice
Prerequisites
Shopping List
System Prep
Install Packages
Creating Containers
Pulling a Container Image
Finding Container Images
Pulling the Container Image
Local Container Images
Running a Container
Running Containers
Custom Images and Containers
Create a Podman Image Registry
Create a Directory for Data to Be Stored
Create Registry Container
Set Podman to Use Insecure Registry
Using the Podman Registry
Tagging Images
Pushing Images
Remote Registries
Customize an Image
Dockerfile
Example
Pull Down CentOS Image
Dockerfile
Build Image
Create Container
Confirm Container Is Running
Delete Container
Container Practices
Cloud Native
Good Practices
Keep It Small
Dynamic Deployment
Scalable
“Does It Cloud”?
Bad Practices
Containers Are Not Virtual Machines
Different Images
Production Builds from Code
Hardcoded Secrets or Configuration
Building Idempotent Containers
Container Development
Development Considerations
Coding Languages
Code Editor
Source Control
Container Tooling
CI/CD
Jenkins Example
Dedicated Image Builders
Image Registry
Development Editor Plugins
Linting Tools
DevSecOps
DevSecOps Tooling
Pipelines
Security Gates
GitOps
GitOps Toolbox
Git
Infrastructure As Code
Pipeline Tools
ArgoCD
Container Orchestration
What Does It Do?
Why Not Use Podman?
Orchestration Options
Kubernetes
Kubernetes Forks
Master Components
The Control Plane
Nodes
Namespaces
Daemonsets
Worker Node Components
Pods
Services
Volumes
Configmaps
OpenShift
Early OpenShift
Current OpenShift
OpenShift Components
Product
Enterprise
Security
Web Console
Many More
Summary
Part III: Day Two Practices and Keeping the Lights On
Chapter 7: Monitoring
Linux Monitoring Tools
Process Monitoring
Default Process Commands, ps and top
Pstree
Resource-Hungry Processes
Memory-Intensive Processes
CPU-Intensive Processes
Disk and IO
iostat and iotop
du and df
CPU
Top
mpstat
Memory
Free
Page Size
Huge Page Size
pmap
Virtual Memory
vmstat
Network
Netstat
ss
iptraf-ng
Tcpdump
NetHogs
iftop
Graphical Tools
Gnome System Monitor
Ksysguard
Historical Monitoring Data
Sar
Performance Co-Pilot
vnstat
Central Monitoring
Nagios
Versions
Core
Nagios XI
Agent Based
NRPE
NRDP
NSClient++
NCPA
Nagios Forks
Installation
Prometheus
Exporters
Alert Tool
Dashboarding
Query Language
Installation
Kubernetes or OpenShift
Configuration
Global
Rule_files
Scrape_configs
Starting Prometheus
Thanos
Sidecar
Store Gateway
Compactor
Receiver
Ruler/Rule
Querier
Query Frontend
Thanos Basic Layout
Enterprise Monitoring
Zabbix
Enterprise Support
Installation
Useful Features
CheckMk
Enterprise Support
Installation
Useful Features
OpenNMS
Enterprise Support
Installation
Useful Features
Dashboards
Dashboarding Tools
Grafana
What Is Grafana
Using Grafana
Cloud Service
On-Premise Installation
Data Sources
Dashboard Creation
Panels
Rows
Save
Application Monitoring
Tracing Tools
Jaeger
Zipkin
Exposing Metrics
How to Speak “Developer”
Summary
Chapter 8: Logging
Linux Logging Systems
Rsyslog
Modular
Installation
Service
Configuration Files
Global Directives
Templates
Rules
Selector Field
Action Field
Fluentd
Plugin Based
Used at Scale
Installation
Prerequisites
Manual Installation
Container Deployment
Configuration
Understanding Logs
Where Are the Log Files
How to Read Log Files
Infrastructure Logs
Important Logs
/var/log/messages
/var/log/secure
/var/log/boot.log
/var/log/dmesg
/var/log/yum.log
/var/log/cron
Application Logs
Good Practice
Use /var/log Directory for Logs
Security
Warn or Above
Increasing Verbosity
Log Verbosity Levels
Log Maintenance
Log Management Tools
Logrotate
Installation
Log Forwarding
Central Logging Systems
Elastic Stack
Fluentd
Log Forwarders
Log Aggregators
Rsyslog
Rsyslog Aggregator
Rsyslog Forwarders
Summary
Chapter 9: Security
Linux Security
Standard Linux Security Tools
Firewall
Iptables
Firewalld
SELinux
Host-Based Intrusion Detection
Recommended Linux Security Configurations
Disable Root Login
Minimal Install
Disk Partitions
Disk Encryption
No Desktop
Encrypt Network Communications
Remove and Disable Insecure or Unused Services
Apply Updates and Patch Kernel
SELinux and Firewall
Improved Authentication Configuration
Check for Open Ports
World Writable Files
Files Not Owned by Anyone
ACLs
Send Logs to Central Logging Service
Intrusion Detection
Application Server Security
DevSecOps
What Is It?
Everyone Is Responsible for Security
Tools
Security Gates
Third-Party Tools
System Compliance
System Hardening
Hardening Standards
Center for Internet Security
Security Technical Implementation Guides
Hardening Linux
Manual Configuration
Automation
OpenSCAP
Vulnerability Scanning
Linux Scanning Tools
OpenVAS
OpenSCAP
ClamAV
Container Image Scanning Tools
Harbor
Role-Based Access
Trivy
Single or Multiple Images
JFrog Xray
Deep Scanning
Clair
Supported Images
Enterprise Version
Continuous Scanning
Dashboard
Pipeline
Container Platform Scanning Tools
Red Hat Advanced Cluster Security for Kubernetes (StackRox)
Vulnerability Scanning
Compliance Scanning
Network Segmentation
Risk Profiling
Configuration Management
Detection and Response
Falco
Flexibly Rules Engine
Immediate Alerting
Current Detection Rules
Aqua Security
Developer Guidance
Informative Dashboarding
Summary
Chapter 10: Maintenance Tasks and Planning
What Maintenance Should Be Done
Patching
Staging
Sandbox
Automated Testing
Automated Patching
Rollback
Filesystem
Cleanup
Check for Errors
Filesystem Check Commands
Firewall
Backups
As Often As Possible
No Live Patching Without Testing
Structure
How Should Maintenance Be Done
Automation
Zero Downtime Environments
Blue/Green
Failover
Maintenance Planning
Agree Maintenance Window
Bite-Size Chunks
Art of Estimating
Automating Process and Task Together
Process Automation
Red Hat PAM
Summary
Part IV: See, Analyze, Then Act
Chapter 11: Troubleshooting
See, Analyze, Then Act
Understand the Problem
Know Where to Start
Standard Questions to Ask When Starting
Explain the Problem
Explain to Yourself
Rubber Duck
Another Person
Use Tools
Break Down the Problem
Onions, They Have Layers
The Five Whys
Example
Theorize Based on Evidence
Hypothesis Building
Build Your Theory
Causality
Prove Your Theory
Reproduce the Issue
Fix in the Test Environment
Remediation
Ask for Help
What to Do Before Asking for Help
Training
How to Ask for Help
Proper Grammar
Spelling
How to Phrase Your Questions
A Better Question
Where to Ask Questions
Correct Area
Forums
GitHub, Stack Overflow
Support Cases
Things to Avoid When Troubleshooting
Live Debugging
Correlation vs. Causation
Being a Lone Wolf
Guessing and Lying
Ghosts
All the Small Things
Keep Track of What You Have Tried
Measure Twice, Cut Once
Do Not Forget Your Retrospective
Summary
Chapter 12: Advanced Administration
System Analysis
Tools for the Sysadmin
Sosreport
xsos
System Information
Shortcut Tools
More Details
System Tracing
Strace
Installation
Output to a File
What to Look For
Useful Strace Parameters
Systemtap
Installation
Manual Install
Automated Install
Systemtap Users
Systemtap Scripts
Running Systemtap Scripts
Cross Instrumentation
System Tuning
Tuned
Installation
Using Tuned
Summary
Index