How to troubleshoot kernel crashes, hangs, or reboots with kdump on Ubuntu Linux
Article ID: 377794
Updated On:
Products
Issue/Introduction
- How do I configure kexec/kdump on Ubuntu systems?
- Root Cause Analysis (RCA) of kernel panic / server crash is required
- How do I troubleshoot and investigate an unexpected reboot?
- How do I generate a kernel memory core dump (vmcore) on my system?
- How do I generate a local kernel crash dump on Ubuntu system?
- Our system entered a hung state or became unresponsive, how can we troubleshoot?
Environment
- Ubuntu Linux
Resolution
Installation
sudo apt install linux-crashdump
Note: Starting with 16.04, the kernel crash dump mechanism is enabled by default.
During the installation, you will be prompted with the following dialogs. Select 'Yes' to enable and select kexec and kump-tools.
|------------------------| Configuring kexec-tools |------------------------|
| |
| |
| If you choose this option, a system reboot will trigger a restart into a |
| kernel loaded by kexec instead of going through the full system boot |
| loader process. |
| |
| Should kexec-tools handle reboots (sysvinit only)? |
| |
| <Yes> <No> |
| |
|---------------------------------------------------------------------------| |------------------------| Configuring kdump-tools |------------------------|
| |
| |
| If you choose this option, the kdump-tools mechanism will be enabled. A |
| reboot is still required in order to enable the crashkernel kernel |
| parameter. |
| |
| Should kdump-tools be enabled be default? |
| |
| <Yes> <No> |
| |
|---------------------------------------------------------------------------|To manually enable the functionality, you can use
dpkg-reconfigure kexec-tools and dpkg-reconfigure kdump-tools commands and answer ‘Yes’ to the questions.
To enable the functionality automatically, you can also edit /etc/default/kexec and set parameters directly:
# Load a kexec kernel (true/false)
LOAD_KEXEC=true
As well, edit /etc/default/kdump-tools to enable kdump by including the following line:
USE_KDUMP=1
Note:
A reboot will be required in order to activate the crashkernel= boot parameter. Upon reboot, kdump-tools will be enabled and active.
You can view the current status of kdump via the command kdump-config show. This will display something like this:
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz
kdump initrd:
/var/lib/kdump/initrd.img
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="..." --initrd=...
Core dumps will be captured under /var/crash directory.
Verification
To confirm that the kernel dump mechanism is opertional, first confirm that the crashkernel boot parameter is present:
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.2.0-17-server root=/dev/mapper/PreciseS-root ro crashkernel=384M-2G:64M,2G-:128M
Now verify that the kernel has reserved the requested memory area for the kdump kernel by running:
dmesg | grep -i crash
...
[ 0.000000] Reserving 64MB of memory at 800MB for crashkernel (System RAM: 1023MB)
Testing crash dump mechanism
Note: After configuring kdump, please schedule down time for the relevant systems in order to manually test a system crash and to verify that a full vmcore is generated in the configured dump target.
- Enable magic key "SysRq" to trigger kernel panic
# echo 1 > /proc/sys/kernel/sysrq
- Trigger SysRq panic event (c) to crash the system manually.
# echo c > /proc/sysrqtrigger
Additional Information
References:
https://ubuntu.com/server/docs/kernel-crash-dump
https://www.kernel.org/doc/Documentation/kdump/kdump.txt
https://www.dedoimedo.com/computers/crash-analyze.html
Feedback
