product iamge

Linux Kernel Debugging: Leverage proven tools and advanced techniques to effectively debug Linux kernels and kernel modules

This document was uploaded by one of our users. The uploader already confirmed that they had the permission to publish it. If you are author/publisher or own the copyright of this documents, please report to us by using this DMCA report form.

Download
Search on Amazon

Simply click on the Download Book button.

Yes, Book downloads on Ebookily are 100% Free.

Sometimes the book is free on Amazon As well, so go ahead and hit "Search on Amazon"

Effectively debug kernel modules, device drivers, and the kernel itself by gaining a solid understanding of powerful open source tools and advanced kernel debugging techniques

Key Features

  • Fully understand how to use a variety of kernel and module debugging tools and techniques using examples
  • Learn to expertly interpret a kernel Oops and identify underlying defect(s)
  • Use easy-to-look up tables and clear explanations of kernel-level defects to make this complex topic easy

Book Description

The Linux kernel is at the very core of arguably the world's best production-quality OS. Debugging it, though, can be a complex endeavor.

Linux Kernel Debugging is a comprehensive guide to learning all about advanced kernel debugging. This book covers many areas in-depth, such as instrumentation-based debugging techniques (printk and the dynamic debug framework), and shows you how to use Kprobes. Memory-related bugs tend to be a nightmare – two chapters are packed with tools and techniques devoted to debugging them. When the kernel gifts you an Oops, how exactly do you interpret it to be able to debug the underlying issue? We've got you covered. Concurrency tends to be an inherently complex topic, so a chapter on lock debugging will help you to learn precisely what data races are, including using KCSAN to detect them. Some thorny issues, both debug- and performance-wise, require detailed kernel-level tracing; you'll learn to wield the impressive power of Ftrace and its frontends. You'll also discover how to handle kernel lockups, hangs, and the dreaded kernel panic, as well as leverage the venerable GDB tool within the kernel (KGDB), along with much more.

By the end of this book, you will have at your disposal a wide range of powerful kernel debugging tools and techniques, along with a keen sense of when to use which.

What you will learn

  • Explore instrumentation-based printk along with the powerful dynamic debug framework
  • Use static and dynamic Kprobes to trap into kernel/module functions
  • Catch kernel memory defects with KASAN, UBSAN, SLUB debug, and kmemleak
  • Interpret an Oops in depth and precisely identify it's source location
  • Understand data races and use KCSAN to catch evasive concurrency defects
  • Leverage Ftrace and trace-cmd to trace the kernel flow in great detail
  • Write a custom kernel panic handler and detect kernel lockups and hangs
  • Use KGDB to single-step and debug kernel/module source code

Who this book is for

This book is for Linux kernel developers, module/driver authors, and testers interested in debugging and enhancing their Linux systems at the level of the kernel. System administrators who want to understand and debug the internal infrastructure of their Linux kernels will also find this book useful. A good grasp on C programming and the Linux command line is necessary. Some experience with kernel (module) development will help you follow along.

Table of Contents

  1. A General Introduction to Debugging Software
  2. Approaches to Kernel Debugging
  3. Debug via Instrumentation – printk and Friends
  4. Debug via Instrumentation – Kprobes
  5. Debugging Kernel Memory Issues – Part 1
  6. Debugging Kernel Memory Issues – Part 2
  7. Oops! Interpreting the Kernel Bug Diagnostic
  8. Lock Debugging
  9. Tracing the Kernel Flow
  10. Kernel Panic, Lockups, and Hangs
  11. Using Kernel GDB (KGDB)
  12. A Few More Kernel Debugging Approaches

Author(s): Kaiwan N Billimoria
Edition: 1
Publisher: Packt Publishing
Year: 2022

Language: English
Commentary: Vector PDF
Pages: 638
City: Birmingham, UK
Tags: Linux; Kernel Debugging; Kernel Modules; Driver Debugging; Kernel GDB; KGDB

Cover
Title Page
Copyright and Credits
Contributors
Table of Contents
Preface
Part 1: A General Introduction and Approaches to Kernel Debugging
Chapter 1: A General Introduction to Debugging Software
Technical requirements
Cloning this book's code repository
Software debugging – what it is, origins, and myths
Software bugs – a few actual cases
Patriot missile failure
The ESA's unmanned Ariane 5 rocket
Mars Pathfinder reset issue
The Boeing 737 MAX aircraft – the MCAS and lack of training of the flight crew
Other cases
Setting up the workspace
Running Linux as a native or guest OS
Running Linux as a guest OS
Installing the Oracle VirtualBox guest additions
Installing required software packages
A tale of two kernels
A production and a debug kernel
Setting up our custom production kernel
Setting up our custom debug kernel
Seeing the difference – production and debug kernel config
Debugging – a few quick tips
A programmer's checklist – seven rules
Summary
Further reading
Chapter 2: Approaches to Kernel Debugging
Technical requirements
Classifying bug types
Types of bugs – the classic view
Types of bugs – the memory view
Types of bugs – the CVE/CWE security-related view
Types of bugs – the Linux kernel
Kernel debugging – why there are different approaches to it
Summarizing the different approaches to kernel debugging
The development phase
Unit testing and/or QA phases
Categorizing into different scenarios
Summary
Further reading
Part 2: Kernel and Driver Debugging Tools and Techniques
Chapter 3: Debug via Instrumentation – printk and Friends
Technical requirements
The ubiquitous kernel printk
Using the printk API's logging levels
Leveraging the pr_ convenience macros
Understanding where the printk output goes
Practically using the printk format specifiers – a few quick tips
Leveraging the printk for debug purposes
Writing debug messages to the kernel log
Debug printing – quick and useful tips
Device drivers – using the dev_dbg() macro
Trying our kernel module on the custom production kernel
Rate limiting the printk
Using the kernel's powerful dynamic debug feature
Dynamic debugging via module parameters
Specifying what and how to print debug messages
Exercising dynamic debugging on a kernel module on a production kernel
Remaining printk miscellany
Printing before console init – the early printk
Designating the printk to some known presets
Printing exactly once
Emitting a printk from userspace
Easily dumping buffer content
Remaining points – bootloader log peeking, LED flashing, and more
Summary
Further reading
Chapter 4: Debug via Instrumentation – Kprobes
Understanding kprobes basics
What we intend to do
Using static kprobes – traditional approaches to probing
Demo 1 – static kprobe – trapping into the file open the traditional static kprobes way – simplest case
Demo 2 – static kprobe – specifying the function to probe via a module parameter
Understanding the basics of the Application Binary Interface (ABI)
Using static kprobes – demo 3 and demo 4
Demo 3 – static kprobe – probing the file open syscall and retrieving the filename parameter
Demo 4 – semi-automated static kprobe via our helper script
Getting started with kretprobes
Kprobes miscellany
Kprobes – limitations and downsides
Interface stability
The easier way – dynamic kprobes or kprobe-based event tracing
Kprobe-based event tracing – minimal internal details
Setting up a dynamic kprobe (via kprobe events) on any function
Using dynamic kprobe event tracing on a kernel module
Setting up a return probe (kretprobe) with kprobe-perf
Trapping into the execve() API – via perf and eBPF tooling
System calls and where they land in the kernel
Observability with eBPF tools – an introduction
Summary
Further reading
Chapter 5: Debugging Kernel Memory Issues – Part 1
Technical requirements
What's the problem with memory anyway?
Tools to catch kernel memory issues – a quick summary
Using KASAN and UBSAN to find memory bugs
Understanding KASAN – the basics
Requirements to use KASAN
Configuring the kernel for Generic KASAN mode
Bug hunting with KASAN
Using the UBSAN kernel checker to find Undefined Behaviour
Building your kernel and modules with Clang
Using Clang 13 on Ubuntu 21.10
Catching memory defects in the kernel – comparisons and notes (Part 1)
Miscellaneous notes
Summary
Further reading
Chapter 6: Debugging Kernel Memory Issues – Part 2
Technical requirements
Detecting slab memory corruption via SLUB debug
Configuring the kernel for SLUB debug
Leveraging SLUB debug features via the slub_debug kernel parameter
Running and tabulating the SLUB debug test cases
Interpreting the kernel's SLUB debug error report
Learning how to use the slabinfo and related utilities
Finding memory leakage issues with kmemleak
Configuring the kernel for kmemleak
Using kmemleak
A few tips for developers regarding dynamic kernel memory allocation
Catching memory defects in the kernel – comparisons and notes (Part 2)
Miscellaneous notes
Summary
Further reading
Chapter 7: Oops! Interpreting the Kernel Bug Diagnostic
Technical requirements
Generating a simple kernel bug and Oops
The procmap utility
What's this NULL trap page anyway?
A simple Oops v1 – dereferencing the NULL pointer
Doing a bit more of an Oops – our buggy module v2
A kernel Oops and what it signifies
The devil is in the details – decoding the Oops
Line-by-line interpretation of an Oops
Tools and techniques to help determine the location of the Oops
Using objdump to help pinpoint the Oops code location
Using GDB to help debug the Oops
Using addr2line to help pinpoint the Oops code location
Taking advantage of kernel scripts to help debug kernel issues
Leveraging the console device to get the kernel log after Oopsing in IRQ context
An Oops on an ARM Linux system and using netconsole
Figuring out the actual buggy code location (on ARM)
A few actual Oopses
Summary
Further reading
Chapter 8: Lock Debugging
Technical requirements
Locking and lock debugging
Locking – a quick summarization of key points
Understanding data races – delving deeper
Catching concurrency bugs with KCSAN
What KCSAN does, in a nutshell
Configuring the kernel for KCSAN
Using KCSAN
Knee-jerk reactions to KCSAN reports – please don't!
A few actual use cases of kernel bugs due to locking defects
Defects identified by KCSAN
Identifying locking rules and bugs from the LDV project
Identifying locking bugs from the Linux kernel Bugzilla
Identifying some locking defects from various blog articles and the like
Summary
Further reading
Part 3: Additional Kernel Debugging Tools and Techniques
Chapter 9: Tracing the Kernel Flow
Technical requirements
Kernel tracing technology – an overview
Using the ftrace kernel tracer
Accessing ftrace via the filesystem
Configuring the kernel for ftrace
Using ftrace to trace the flow of the kernel
Useful ftrace filtering options
Case 1 – tracing a single ping with raw ftrace
Case 2 – tracing a single ping with raw ftrace via the set_event interface
Using trace_printk() for debugging
Ftrace – miscellaneous remaining points via FAQs
Ftrace use cases
Using the trace-cmd, KernelShark, and perf-tools ftrace frontends
An introduction to using trace-cmd
Using the KernelShark GUI
An introduction to using perf-tools
An introduction to kernel tracing with LTTng and Trace Compass
A quick introduction to recording a kernel tracing session with LTTng
Using the Trace Compass GUI to visualize the single ping LTTng trace
Summary
Further reading
Chapter 10: Kernel Panic, Lockups, and Hangs
Technical requirements
Panic! – what happens when a kernel panics
Let's panic
To the rescue with netconsole
Interpreting the panic output
Kernel parameters, tunables, and configs that affect kernel panic
Writing a custom kernel panic handler routine
Linux kernel panic notifier chains – the basics
Setting up our custom panic handler within a module
Detecting lockups and CPU stalls in the kernel
A short note on watchdogs
Employing the kernel's hard and soft lockup detector
Employing the kernel's hung task and workqueue stall detectors
Leveraging the kernel hung task detector
Detecting workqueue stalls
Summary
Further reading
Chapter 11: Using Kernel GDB (KGDB)
Technical requirements
Conceptually understanding how KGDB works
Setting up an ARM target system and kernel for KGDB
Building a minimal custom ARM Linux target system with SEALS
Configuring the kernel for KGDB
Testing the target system
Debugging the kernel with KGDB
Running our target (emulated) ARM32 system
Running and working with the remote GDB client on the host system
Debugging kernel modules with KGDB
Informing the GDB client about the target module's locations in memory
Step by step – debugging a buggy module with KGDB
[K]GDB – a few tips and tricks
Setting up and using GDB scripts with CONFIG_GDB_SCRIPTS
KGDB target remote :1234 command doesn't work on physical systems
Setting the system root with sysroot
Using GDB's TUI mode
What to do when the GDB response occurs
GDB convenience routines
GDB custom macros in its startup file
Fancy breakpoints and hardware watchpoints
Miscellaneous GDB tips
Summary
Further reading
Chapter 12: A Few More Kernel Debugging Approaches
An introduction to the kdump/crash framework
Why use kdump/crash?
Understanding the kdump/crash basic framework
A mention on performing static analysis on kernel code
Examples using cppcheck and checkpatch.pl for static analysis
An introduction to kernel code coverage tools and testing frameworks
Why is code coverage important?
A brief note on kernel testing
Miscellaneous – using journalctl, assertions, and warnings
Looking up system logs with journalctl
Assertions, warnings, and BUG() macros
Summary
Further reading
Index
Other Books You May Enjoy