Logging in Action: With Fluentd, Kubernetes and more

Logging in Action
contents
foreword
preface
acknowledgments
about this book
Who should read this book
How this book is organized
About the code
liveBook discussion forum
about the author
about the cover illustration
Part 1 From zero to “Hello World”
1 Introduction to Fluentd
1.1 Elevator pitch for Fluentd
1.1.1 What is a log event?
1.1.2 Fluentd compared to middleware
1.2 Why do we produce logs?
1.3 Evolving ideas
1.3.1 Four golden signals
1.3.2 Three pillars of observability
1.4 Log unification
1.4.1 Unifying logs vs. log analytics
1.5 Software stacks
1.5.1 ELK stack
1.5.2 Comparing Fluentd and Logstash
1.5.3 The relationship between Fluentd and Fluent Bit
1.5.4 The relationship between Logstash and Beats
1.6 Log routing as a vehicle for security
1.7 Log event life cycle
1.8 Evolution of Fluentd
1.8.1 Treasure Data
1.8.2 CNCF
1.8.3 Relationship to major cloud vendors PaaS/IaaS
1.9 Where can Fluentd and Fluent Bit be used?
1.9.1 Platform constraints
1.10 Fluentd UI-based editing
1.11 Plugins
1.12 How Fluentd can be used to make operational tasks easier
1.12.1 Actionable log events
1.12.2 Making logs more meaningful
1.12.3 Polyglot environments
1.12.4 Multiple targets
1.12.5 Controlling log data costs
1.12.6 Logs to metrics
1.12.7 Rapid operational consolidation
Summary
2 Concepts, architecture, and deployment of Fluentd
2.1 Architecture and core concepts
2.1.1 The makeup of a log event
2.1.2 Handling time
2.1.3 Architecture of Fluentd
2.1.4 Fluent configuration execution order
2.1.5 Directives
2.1.6 Putting timing requirements into action
2.2 Deployment of Fluentd
2.2.1 Deploying Fluentd for the book’s examples
2.2.2 Deployment considerations for Fluentd
2.2.3 Fluentd minimum footprint
2.2.4 Simple deployment of Ruby
2.2.5 Simple deployment of Fluentd
2.2.6 Deploying a log generator
2.2.7 Installing Postman
2.3 Bringing Fluentd to life with “Hello World”
2.3.1 “Hello World” scenario
2.3.2 “Hello World” configuration
2.3.3 Starting Fluentd
2.4 “Hello World” with Fluent Bit
2.4.1 Starting Fluent Bit
2.4.2 Alternate Fluent Bit startup options
2.4.3 Fluent Bit configuration file comparison
2.4.4 Fluent Bit configuration file in detail
2.4.5 Putting the dummy plugin into action
2.5 Fluentd deployment with Kubernetes and containers
2.5.1 Fluentd DaemonSet
2.5.2 Dockerized Fluentd
2.6 Using Fluentd UI
2.6.1 Installing Fluentd with UI
Summary
Part 2 Fluentd in depth
3 Using Fluentd to capture log events
3.1 Dry running to check a configuration
3.1.1 Putting validating Fluentd configuration into action
3.2 Reading log files
3.2.1 Putting the adaption of a Fluentd configuration to Fluent Bit into action
3.2.2 Rereading and resuming reading of log files
3.2.3 Configuration considerations for tracking position
3.2.4 Wildcards in the path attribute
3.2.5 Expressing time
3.2.6 Controlling the impact of wildcards in filenames
3.2.7 Replacing wildcards with delimited lists in action
3.2.8 Handling log rotation
3.3 Self-monitoring
3.3.1 HTTP interface check
3.4 Imposing structure on log events
3.4.1 Standard parsers
3.4.2 Third-party parsers
3.4.3 Applying a Regex parser to a complex log
3.4.4 Putting parser configuration into action
Summary
4 Using Fluentd to output log events
4.1 File output plugin
4.1.1 Basic file output
4.1.2 Basics of buffering
4.1.3 Chunks and Controlling Buffering
4.1.4 Retry and backoff
4.1.5 Putting configuring buffering size settings into action
4.2 Output formatting options
4.2.1 out_file
4.2.2 json
4.2.3 ltsv
4.2.4 csv
4.2.5 msgpack
4.2.6 Applying formatters
4.2.7 Putting JSON formatter configuration into action
4.3 Sending log events to MongoDB
4.3.1 Deploying MongoDB Fluentd plugin
4.3.2 Configuring the Mongo output plugin for Fluentd
4.3.3 Putting MongoDB connection configuration strings into action
4.4 Actionable log events
4.4.1 Actionable log events through service invocation
4.4.2 Actionable through user interaction tools
4.5 Slack to demonstrate the social output
4.5.1 Handling tokens and credentials more carefully
4.5.2 Externalizing Slack configuration attributes in action
4.6 The right tool for the right job
Summary
5 Routing log events
5.1 Reaching multiple outputs by copying
5.1.1 Copy by reference or by value
5.1.2 Handling errors when copying
5.2 Configuration reuse and extension through inclusion
5.2.1 Place holding with null output
5.2.2 Putting inclusions with a MongoDB output into action
5.3 Injecting context into log events
5.3.1 Extraction of values
5.4 Tag-based routing
5.4.1 Using exec output plugin
5.4.2 Putting tag naming conventions into action
5.4.3 Putting dynamic tagging with extract into action
5.5 Tag plugins
5.6 Labels: Taking tags to a new level
5.6.1 Using a stdout filter to see what is happening
5.6.2 Illustrating label and tag routing
5.6.3 Connecting pipelines
5.6.4 Label sequencing
5.6.5 Special labels
5.6.6 Putting a common pipeline into action
Summary
6 Filtering and extrapolation
6.1 Application of filters
6.1.1 All is well events do not need to be distributed
6.1.2 Spotting the needle in a haystack
6.1.3 False urgency
6.1.4 Releveling
6.1.5 Unimplemented housekeeping
6.2 Why change log events?
6.2.1 Easier to process meaning downstream
6.2.2 Add context
6.2.3 Record when we have reacted to a log event
6.2.4 Data redaction/masking
6.3 Applying filters and parsers
6.3.1 Filter plugins
6.3.2 Applying grep filters
6.3.3 Changing log events with the record_transformer plugin
6.3.4 Filter parser vs. record transformer
6.4 Demonstrating change impact with stdout in action
6.4.1 A solution demonstrating change impact with stdout in action
6.5 Extract to set key values
6.6 Deriving new data values with the record_transformer
6.6.1 Putting the incorporation of calculations into a log event transformation into action
6.7 Generating simple Fluentd metrics
6.7.1 Putting log event counting into action
Summary
Part 3 Beyond the basics
7 Performance and scaling
7.1 Threading and processes to scale with workers
7.1.1 Seeing workers in action
7.1.2 Worker constraints
7.1.3 Controlling output plugin threads
7.1.4 Memory management optimization
7.2 Scaling and moving workloads
7.2.1 Fan-in/log aggregation and consolidation
7.2.2 Fan-out and workload distribution
7.2.3 High availability
7.2.4 Putting a high-availability comparison into action
7.3 Fluentd scaling in containers vs. native and virtual environments
7.3.1 Kubernetes worker node configuration
7.3.2 Per-cluster configuration
7.3.3 Container as virtualization
7.3.4 Sidecar pattern
7.3.5 Options comparison
7.4 Securing traffic between Fluentd nodes
7.4.1 TLS configuration
7.4.2 TLS not just for encryption
7.4.3 Certificate and private key storage
7.4.4 Security is more than certificates
7.5 Credentials management
7.5.1 Simple credentials use case
7.5.2 Putting certification into action
Summary
8 Driving logs with Docker and Kubernetes
8.1 Fluentd out of the box from Docker Hub
8.1.1 Official Docker images
8.1.2 Docker log drivers
8.1.3 Getting set up for Docker log drivers
8.2 Using Docker log drivers
8.2.1 Docker drivers via the command line
8.2.2 A quick check of network connections
8.2.3 Running Docker command line
8.2.4 Switching to driver configuration through a configuration file
8.3 Kubernetes components logging and the use of Fluentd
8.3.1 Kubernetes components and structured logging
8.3.2 Kubernetes default log retention and log rotation
8.3.3 kubectl with logging
8.4 Demonstrating logging with Kubernetes
8.4.1 Kubernetes setup
8.4.2 Creating logs to capture
8.4.3 Understanding how Fluentd DaemonSets are put together
8.5 Getting a peek at host logs
8.6 Configuring a Kubernetes logging DaemonSet
8.6.1 Getting the Fluentd configuration ready to be used
8.6.2 Creating our Kubernetes deployment configuration
8.6.3 Putting the implementation of a Fluentd for Kubernetes into action
8.6.4 Deploying to minikube
8.6.5 Tidying up
8.7 Kubernetes configuration in action
8.7.1 Answer
8.8 More Kubernetes monitoring and logging to watch for
8.8.1 Node monitoring
8.8.2 Termination messages
Summary
9 Creating custom plugins
9.1 Plugin source code
9.2 What is Redis, and why build a plugin with the Redis list capability?
9.2.1 Redis list over RedisTimeSeries
9.3 Illustrating our objective using Redis CLI
9.4 Preparing for development
9.5 Plugin frameworks
9.5.1 Creating the skeleton plugin
9.5.2 Plugin life cycle
9.6 Implementing the plugin core
9.6.1 How configuration attributes work
9.6.2 Starting up and shutting down
9.6.3 Getting the plugin to work with our Fluentd installation
9.6.4 Putting additional configuration validation into action
9.6.5 Implementing the Redis output logic
9.6.6 Putting the testing of synchronous output into action
9.7 Implementing the Redis input plugin
9.7.1 Testing input and output plugin execution
9.8 Extending output with buffering
9.8.1 Improving our scenario by putting maintainability into action
9.9 Unit testing
9.10 Putting the development of unit tests into action
9.10.1 Answer
9.11 Package and deployment
9.11.1 Documentation
9.11.2 Complete metadata aka manifest
9.11.3 Building the gem package
9.11.4 Rerun without the plugin paths
9.12 Extending to be an enterprise-class solution
Summary
Part 4 Good logging practices and frameworks to maximize log value
10 Logging best practices
10.1 Audit events vs. log events
10.2 Log levels and severities
10.2.1 Trace
10.2.2 Debug
10.2.3 Info(rmation)
10.2.4 Warn(ing)
10.2.5 Error
10.2.6 Fatal
10.2.7 Extending or creating your own log levels
10.3 Clear language
10.4 Human and machine-readable
10.5 Context is key
10.5.1 Context: What
10.5.2 Context: When
10.5.3 Context: where
10.5.4 Context: Why
10.5.5 Context: Who
10.5.6 a practical checklist for capturing context
10.6 Error codes
10.6.1 Using standard errors
10.6.2 Codes can be for more than errors
10.7 Too little logging or too much?
10.7.1 What qualifies as sensitive?
10.7.2 GDPR is only the start
10.8 Log structure and format
10.8.1 Putting making log entries ready for application shipping into action
10.9 Use frameworks if you can
10.10 Development practices
10.10.1 Rethrowing exceptions
10.10.2 Using standard exceptions and error structures
10.10.3 String construction as a reason not to log
Summary
11 Logging frameworks
11.1 Value of logging frameworks
11.2 Typical structure of a logging framework
11.2.1 Logger context
11.2.2 Appender
11.2.3 Logger
11.2.4 Filter
11.2.5 Formatter
11.2.6 Configuration
11.2.7 Logger config
11.3 Appender structures
11.4 Logging framework landscape
11.5 Choosing a framework
11.5.1 Putting optimizing application logging into action
11.6 Fluentd’s own logging and appenders
11.7 Illustrations of an application logging directly to Fluentd
11.7.1 Python with logging framework: Using the Fluentd library
11.7.2 Invoking Fluentd appender directly
11.7.3 Illustration with only Python’s logging
11.7.4 Illustration without Python’s logging or Fluentd library
11.7.5 Porting the Fluentd calls to another language into action
11.7.6 Using generic appenders: The takeaways
Summary
appendix A Processing times and dates, regular expressions, and other configuration values
A.1 Tool installation overview
A.2 Creating environment variables and amending PATH
A.2.1 Windows
A.2.2 Linux
A.3 Java and Groovy
A.4 Postman
A.5 Elasticsearch
A.5.1 Core Elasticsearch installation
A.5.2 Elasticsearch UI installation
A.5.3 Fluentd plugin for Elasticsearch
A.6 Mongo database
A.6.1 Mongo DB installation
A.6.2 MongoDB configuration
A.7 Slack
A.8 Setting up Docker and Kubernetes
A.8.1 Windows Prerequisites
A.8.2 VirtualBox approach
A.8.3 Ubuntu image preparation for working with Docker
A.8.4 Kubernetes installation
A.9 Support Ruby development libraries and tools
A.10 Redis
A.10.1 Redis gem
A.11 Python
A.12 Vault
appendix B Processing times and dates, regular expressions, and other configuration values
B.1 Expressing relative time
B.2 Expressing dates and times
B.3 Expressing sizes
B.4 Regular expressions
B.4.1 Escape Codes
B.4.2 Repetition/selection
B.4.3 Anchors, groups, and alternates
B.5 Docker tag customization
appendix C Plugins summary
C.1 Formatter plugins
C.2 Extract and inject plugin support
C.3 Filter plugins
C.4 Tag manipulation plugins
C.5 Preventing alert storms
C.6 Analytical and metrics plugins
C.7 Plugin Interfaces
appendix D Real-world use case
D.1 Fluentd use in a real-world use case
D.2 Setting the scene
D.2.1 The operational challenges
D.3 Introducing monitoring
D.3.1 Extending monitoring
D.3.2 Finessing of monitoring
D.4 Cloud dilemma
D.5 Solution
D.6 Conclusion
appendix E Useful resources
E.1 Helpful Fluentd resources
E.2 Helpful Fluentd third-party tools
E.3 Helpful logging practices resources
E.4 Common log formats and descriptions
E.5 Helpful Ruby resources
E.6 Docker and Kubernetes
E.7 Elasticsearch
E.8 Redis
E.9 SSL/TLS and security
E.10 Environment setup
E.11 Logging frameworks
E.12 Information portals on legislation
E.13 Other handy sources of information
E.14 Supporting Fluentd resources
E.15 Related reading
index
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
R
S
T
U
V
W
X
Y
Logging in Action - back