"[Linux] Generate kernel crash dump"

Steps

  1. Make sure following configs are on/off for current kernel
# CONFIG_KASAN is not set
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_DEBUG_INFO=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_PROC_VMCORE=y
  1. Install kexec and crash tool
$ sudo apt install kexec-tools makedumpfile crash
  1. Add 'crashkernel' parameter to grub
$ sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="crashkernel=512M"
$ sudo update-grub
  1. Load new kernel to be rebooted to into memory
$ sudo kexec -p -l /boot/vmlinuz-$(KDUMP_KVER) \
       --append="$(cat /proc/cmdline | sed 's/crashkernel=[^\s]*//') nr_cpus=1 irqpoll" \
       --initrd=/boot/initrd.img-$(KDUMP_KVER)
  1. Trigger kernel panic and new kernel will be automatically rebooted to
$ echo c | sudo tee /proc/sysrq-trigger
  1. Generate compressed kdump file from /proc/vmcore
(In new kernel)
$ sudo makedumpfile -c -d 31
       -x /usr/lib/debug/boot/vmlinux-$(PANIC_KVER) /proc/vmcore dumpfile
  1. Use crash tool to inspect generated kdump file
$ sudo crash /usr/lib/debug/boot/vmlinux-$(PANIC_KVER) dumpfile

Note

$ sudo apt install bison
$ git clone https://github.com/crash-utility/crash.git
$ cd crash
$ aria2c -s 16 -x 16 http://mirrors.nju.edu.cn/gnu/gdb/gdb-7.6.tar.gz
$ tar xvf gdb-7.6.tar.gz
$ make -j16
$ sudo make install

$ aria2c -s 16 -x16 https://sourceforge.net/projects/makedumpfile/files/makedumpfile/1.6.7/makedumpfile-1.6.7.tar.gz/download
$ tar xvf makedumpfile-1.6.7.tar.gz
$ cd makedumpfile-1.6.7
(refer README for build and intall steps)

kernel 5.5 + makedumpfile 1.6.7 + crash 7.2.8

References

[1] https://github.com/crash-utility/crash
[2] https://sourceforge.net/projects/makedumpfile
[3] https://help.ubuntu.com/lts/serverguide/kernel-crash-dump.html