product iamge

Linux Kernel Debugging

This document was uploaded by one of our users. The uploader already confirmed that they had the permission to publish it. If you are author/publisher or own the copyright of this documents, please report to us by using this DMCA report form.

Download
Search on Amazon

Simply click on the Download Book button.

Yes, Book downloads on Ebookily are 100% Free.

Sometimes the book is free on Amazon As well, so go ahead and hit "Search on Amazon"

Debug Linux kernel modules and the kernel itself by mastering powerful open source tools and advanced kernel debugging techniques

Key Features

  • Learn how to use a variety of kernel and module debug tools and techniques with the help of examples
  • Understand how to expertly interpret a kernel oops and identify the underlying defects
  • Includes easy-to-look up tables and simplified explanations of complex kernel-level defects

Book Description

The Linux kernel is at the very core of arguably the world’s best production-quality OS. Debugging it, though, can be a complex endeavor. Linux Kernel Debugging is a thorough guide to advanced kernel debugging.

This book covers a range of debugging techniques in detail, starting with instrumentation-based debugging techniques, the kernel's powerful Kprobes framework, printk, and friends. The book features two whole chapters devoted to the tools and techniques for debugging memory-related bugs. It also prepares you to interpret the underlying issue when the Linux kernel throws an oops. As you advance through the chapters, you'll be able to demystify concurrency by learning what data races are and how to handle them, including the use of modern features like Kernel Concurrency Sanitizer (KCSAN). You'll also discover how to overcome some thorny issues related to debugging and performance through detailed kernel-level tracing and learn to wield the power of Ftrace and its frontends. What's more? The book shows you how to deal with kernel hangs and panic, leverage the venerable GDB tool within the kernel, and much more.

By the time you complete reading this debugging kernel book, you'll have at your disposal, a whole range of powerful debug tools and techniques, along with a keen sense as to when to use which.

What you will learn

  • Instrumentation-based printk and the powerful dynamic debug framework
  • Use Kprobes statically and dynamically to trap into kernel or module functions
  • Catch kernel memory defects with KASAN, UBSAN, SLUB debug, and kmemleak
  • Precisely identify the source location of kernel and module bugs
  • Understand data races and use KCSAN to catch evasive concurrency defects
  • Leverage Ftrace and trace-cmd to trace the kernel flow in great detail
  • Write a custom kernel panic handler using the WD and hangcheck timer
  • Learn to use KGDB to single-step and debug kernel or module source code

Who This Book Is For

This book is for Linux kernel developers, module or driver authors, and testers interested in debugging and enhancing their Linux systems at the level of the kernel. System administrators who want to understand and debug the internal infrastructure of their Linux kernels will also find this book useful. A good hold of C programming and the Linux command line is necessary. Some experience with module or kernel development will be beneficial too.

Table of Contents

  1. A General Introduction to Debugging Software
  2. Approaches to Kernel Debugging
  3. Debugging via Instrumentation - printk and friends
  4. Debugging via Instrumentation – Using Kprobes
  5. Debugging Kernel Memory Issues – Part 1
  6. Debugging Kernel Memory Issues – Part 2
  7. Oops! Interpreting the Kernel Bug Diagnostic
  8. Lock Debugging
  9. Tracing the Kernel Flow
  10. Kernel Panic, Hangcheck, and Watchdogs
  11. Using KGDB
  12. Other Approaches to Kernel Debugging

Author(s): Kaiwan N Billimoria
Edition: 1
Publisher: Packt Publishing
Year: 2022

Language: English
Pages: 571
Tags: Linux; Kernel Debugging

Table of Contents

Linux Kernel Debugging
Linux Kernel Debugging
1 A General Introduction to Debugging Software
Technical requirements
Cloning this book’s code repository
Software debugging – what it is, origins, and myths
Software bugs – a few actual cases
Patriot missile failure
ESA’s unmanned Ariane 5 rocket
Mars Pathfinder reset issue
The Boeing 737 MAX aircraft – the MCAS and lack of training to the flight crew
Other cases
Setting up the workspace
Running Linux as a native or guest OS?
Running Linux as a guest OS
Installing the Oracle VirtualBox guest additions
Installing required software packages
A tale of two kernels
A production and a debug kernel
Setting up our custom production kernel
Setting up our custom debug kernel
Seeing the difference – production and debug kernel config
Debugging – a few quick tips
A programmer’s checklist – seven rules
Summary
Further reading
2 Approaches to Kernel Debugging
Technical requirements
Classifying bug types
Types of bugs – the classic view
Types of bugs – the memory view
Types of bugs – the CVE/CWE security-related view
Types of bugs – the Linux kernel
Kernel debugging – why there are different approaches to it
Summarizing the different approaches to kernel debugging
Development phase
Unit testing and/or QA phases
Categorizing into different scenarios
Summary
Further reading
3 Debug via Instrumentation – printk and friends
Technical requirements
The ubiquitous kernel printk
Using the printk API’s logging levels
Leveraging the pr_ convenience macros
Understanding where the printk output goes
Practically using the printk format specifiers – a few quick tips
Leveraging the printk for debug purposes
Writing debug messages to the kernel log
Debug printing – quick and useful tips
Device drivers – use the dev_dbg()
Trying our kernel module on the custom production kernel
Rate limiting the printk
Using the kernel’s powerful dynamic debug feature
Dynamic debug via module parameters
Specifying what and how to print debug messages
Exercising dynamic debug on a kernel module on a production kernel
Remaining printk miscellany
Printing before console init – the early printk
Designating the printk to some known presets
Printing exactly once
Emitting a printk from userspace
Easily dumping buffer content
Remaining points – bootloader log peeking, LED flashing and more
Summary
Further reading
4 Debug via Instrumentation – Kprobes
Understanding kprobes basics
What we intend to do
Using static kprobes – traditional approaches to probing
Demo 1 – static kprobe – trapping into the file open the traditional static kprobes way – simplest case
Demo 2 – Static kprobe – specifying the function to probe via a module parameter
Understanding the basics of the Application Binary Interface (ABI)
Using static kprobes – demo 3 and demo 4
Demo 3 – static kprobe – probing the file open and retrieving the filename parameter
Demo 4 – Semi-automated static kprobe via our helper script
Getting started with kretprobes
Kprobes miscellany
Kprobes – limitations and downsides
Interface stability
The easier way – dynamic kprobes or kprobe-based event tracing
Kprobe-based event tracing – minimal internal details
Setting up a dynamic kprobe (via kprobe events) on any function
Using dynamic kprobe event tracing on a kernel module
Setting up a return probe (kretprobe) with kprobe-perf
Trapping into the execve() – via perf and eBPF tooling
System calls and where they land in the kernel
Observability with eBPF tools – an introduction
Summary
Further reading
5 Debugging Kernel Memory Issues – Part 1
Technical requirements
What’s the problem with memory anyway?
Tools to catch kernel memory issues – a quick summary
Using KASAN and UBSAN to find memory bugs
Understanding KASAN – the basics
Requirements to use KASAN
Configuring the kernel for generic KASAN mode
Bug hunting with KASAN
Using the UBSAN kernel checker to find UB
Building your kernel and modules with Clang
Using Clang 13 on Ubuntu 21.10
Catching memory defects in the kernel – comparisons and notes (Part 1)
Miscellaneous notes
Summary
Further reading
6 Debugging Kernel Memory Issues – Part 2
Technical requirements
Detecting slab memory corruption via SLUB debug
Configuring the kernel for SLUB debug
Leveraging SLUB debug features via the slub_debug kernel parameter
Running and tabulating the SLUB debug test cases
Interpreting the kernel’s SLUB debug error report
Learning to use the slabinfo and related utilities
Finding memory leakage issues with kmemleak
Configuring the kernel for kmemleak
Using kmemleak
A few tips for developers regarding dynamic kernel memory allocation
Catching memory defects in the kernel – comparisons and notes (Part 2)
Miscellaneous notes
Summary
Further reading
7 Oops! Interpreting the kernel bug diagnostic
Technical requirements
Generating a simple kernel bug and Oops
The procmap utility
What’s this NULL trap page anyway
A simple Oops v1 – dereferencing the NULL pointer
Doing a bit more of an Oops – our buggy module v2
A kernel Oops and what it signifies
Devil in the details – decoding the Oops
Line-by-line interpretation of an Oops
Tools and techniques to help determine the location of the Oops
Using objdump to help pinpoint the Oops code location
Using GDB to help debug the Oops
Using addr2line to help pinpoint the Oops code location
Taking advantage of kernel scripts to help debug kernel issues
Leveraging the console device to get the kernel log after Oops’ing in IRQ context
An Oops on an ARM Linux system and using netconsole
Figuring out the actual buggy code location (on ARM)
A few actual Oops’es
Summary
Further reading