A Minimal, Debuggable Linux Kernel with QEMU and BusyBox

1113 words
6 minutes
A Minimal, Debuggable Linux Kernel with QEMU and BusyBox
This post is also available in 简体中文.

I wrote this in 2021 and redid it in 2026. The kernel moved from 4.9 to a 6.x that’s still maintained, the root filesystem changed from an ext3 image to a cpio initramfs (no sudo mount, no root needed), and I added the step I never wrote back then, how to actually debug into it with GDB. This setup is still the standard starting point for reading the kernel and reproducing kernel bugs today.

What you need#

  • Ubuntu 22.04 or newer (anything Debian-based works the same)
  • Linux 6.x source
  • The latest stable BusyBox
  • QEMU

Getting the source and tools#

Grab the kernel source from Kernel.org and BusyBox from the BusyBox site:

Terminal window
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.tar.xz
wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
tar xf linux-6.6.tar.xz
tar xf busybox-1.36.1.tar.bz2
sudo apt install qemu-system-x86 build-essential flex bison libelf-dev libssl-dev bc cpio

Back then I used 4.9 + BusyBox 1.33, and I couldn’t even keep the version straight (1.33.0 in one place, 1.33.1 in another). It’s consistent below.

Building the kernel#

Terminal window
cd linux-6.6
make defconfig # or make x86_64_defconfig
make menuconfig

In menuconfig, make sure of a few things (defconfig has most of them already):

  • General setup -> Initial RAM filesystem and RAM disk (initramfs/initrd) support
  • Device Drivers -> Block devices -> RAM block device support (with a cpio initramfs you don’t really need this one, but leaving it on doesn’t hurt)
  • Kernel hacking -> Compile-time checks and compiler options -> Compile the kernel with debug info, which you must turn on for debugging
  • If you want GDB breakpoints to behave nicely, turn on Kernel hacking -> Compile the kernel with frame pointers too

If this step fails with fatal error: curses.h, install libncurses-dev (called libncurses5-dev on older releases).

Then build:

Terminal window
make -j$(nproc)

The built kernel is at arch/x86/boot/bzImage. My old notes weren’t consistent here, saying arch/x86/boot in one place and arch/x86_64/boot in another. The x86_64 kernel is also at arch/x86/boot/bzImage, and there’s no bzImage in arch/x86_64/boot/. I fell into that one back then.

Building BusyBox#

Terminal window
cd ../busybox-1.36.1
make defconfig
make menuconfig
  • Settings -> Build Options -> [*] Build static binary (no shared libs)

There’s a trap when building BusyBox on newer systems. The tc applet just won’t compile, and you get a pile of errors like TCA_CBQ_MAX undeclared and incomplete type 'struct tc_cbq_lssopt'. The kernel headers dropped that batch of old CBQ queueing constants in 5.x, but BusyBox’s tc still uses them. The fix is to turn tc off in menuconfig, under Networking Utilities -> [ ] tc. This is also where I got stuck back then.

Terminal window
make -j$(nproc)
make install

Everything it installs ends up in _install/.

Making the root filesystem (cpio initramfs)#

Back then I loop-mounted an ext3 image and copied files into it, with sudo. No need anymore. The kernel supports cpio initramfs natively, so you just pack it up:

Terminal window
mkdir -p rootfs/{bin,sbin,etc/init.d,proc,sys,dev,tmp,mnt}
cp -a _install/* rootfs/

rootfs/etc/init.d/rcS:

rootfs/etc/init.d/rcS
#!/bin/sh
echo "Welcome to my little Linux"
/bin/mount -a
mkdir -p /dev/pts
mount -t devpts devpts /dev/pts
echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

rootfs/etc/fstab:

rootfs/etc/fstab
proc /proc proc defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0

rootfs/etc/inittab:

rootfs/etc/inittab
::sysinit:/etc/init.d/rcS
::respawn:-/bin/sh
::askfirst:-/bin/sh
::ctrlaltdel:/bin/umount -a -r

These files had two typos back then, both fixed now. moutn is now mount, and cttlaltdel is now ctrlaltdel (a misspelled inittab entry doesn’t do anything, but it doesn’t complain either, so Ctrl+Alt+Del just quietly does nothing).

If the device nodes aren’t in the archive, mdev -s creates them after boot. To be safe, make two up front:

Terminal window
sudo mknod rootfs/dev/console c 5 1
sudo mknod rootfs/dev/null c 1 3
chmod +x rootfs/etc/init.d/rcS rootfs/etc/inittab

Pack it:

Terminal window
cd rootfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../initramfs.cpio.gz

Booting it in QEMU#

Terminal window
qemu-system-x86_64 \
-kernel linux-6.6/arch/x86/boot/bzImage \
-initrd initramfs.cpio.gz \
-append "console=ttyS0 rdinit=/sbin/init" \
-nographic

When the kernel prints its boot log and you end up at a /bin/sh prompt, you’re done.

A couple of options are worth remembering. console=ttyS0 points the console at the serial port, so with -nographic you can work right in your terminal without opening a graphical window. rdinit= names the first program in the initramfs, which becomes PID 1.

I hit a trap here while redoing this. At first I pointed rdinit straight at /etc/init.d/rcS. The kernel did run the script and print the welcome message, and then it panicked with Attempted to kill init! exitcode=0x00000000. The script is PID 1, and when it finishes it exits, and once PID 1 exits the kernel has nothing left to run, so it can only crash. That’s why rdinit should be /sbin/init (BusyBox’s init). It reads /etc/inittab, runs rcS from the ::sysinit: line, then starts a shell from the ::askfirst: line, so PID 1 stays around. For the same reason, renaming the script to /init doesn’t help either, unless the script ends with exec /bin/sh to turn itself into a process that sticks around.

Attaching GDB to the kernel#

This is the step I didn’t write back then, and it’s actually the most useful thing about this setup. Add two options to QEMU:

Terminal window
qemu-system-x86_64 \
-kernel linux-6.6/arch/x86/boot/bzImage \
-initrd initramfs.cpio.gz \
-append "console=ttyS0 rdinit=/sbin/init nokaslr" \
-nographic -s -S
  • -s is short for -gdb tcp::1234, which opens a gdbstub on port 1234
  • -S pauses the CPU right at startup and waits for the debugger to send continue
  • nokaslr matters a lot. With KASLR on, the kernel’s addresses change on every boot, and none of the breakpoint addresses you set in GDB will line up. Turn randomization off while you’re studying the kernel

In another terminal:

Terminal window
gdb linux-6.6/vmlinux
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) continue
(gdb) list
(gdb) bt

vmlinux is the kernel image with symbols that the build produces (in the source root), not bzImage. hbreak sets a hardware breakpoint. Early in boot, before the kernel has set up its page tables, software breakpoints aren’t reliable, so a hardware breakpoint is the safer first stop. Once you’re in, plain b works fine.

Looking at it again today#

This setup has barely changed in twenty years, because it’s just what kernel developers do every day: build, start QEMU, attach GDB. What changed is the details. The root filesystem went from an ext3 image to cpio (which saves you the whole root and loop mount dance), debugging went from a serial port to a gdbstub, the kernel went from 4.9 to 6.x, and the option names in menuconfig have been renamed a few times.

If you want to read the kernel, this setup is honestly great value. Being able to set breakpoints, look at the stack and step through the scheduler or a filesystem makes things click much faster than just chewing through source. If you want to reproduce kernel bugs (the CVE kind), this is pretty much the standard starting point. Find an affected version, build it the same way, add qemu + gdb and a bit of KASAN, and you can see the crash clearly on your own machine. That’s also why this post was worth keeping from 2021 until today.

Share Article

If this article helped you, please share it with others!

A Minimal, Debuggable Linux Kernel with QEMU and BusyBox
https://blog.lumysia.net/posts/minimal-linux-kernel-qemu-busybox/
Author
Livia
Published at
2021-07-07

Comments

Table of Contents