How to enable a Kernel Crash Dump in Airgapped VKS Clusters on Ubuntu

Products

• Tanzu Kubernetes Runtime
• VMware vSphere Kubernetes Service

Issue/Introduction

This knowledge base article documents the steps to enable kernel crash dump on VKS clusters in an airgapped environment.

These steps involve assessing the full set of package dependencies required through a VM that does have internet access and proceeding with steps based on the gathered information.

Environment

vSphere Supervisor

Airgapped environment

VKS Cluster Nodes running on Ubuntu OS

VKR v1.35.2 and lower

Resolution

Collect Package Dependencies - VM with Internet Access

1. Document which Ubuntu OS version is in use.
   Note: It is important to ensure that package dependencies consistently use the same Ubuntu OS version for the following steps.
   
   
2. Connect to a VM with internet access on the same Ubuntu OS version documented above
   
   
3. Create a script using the contents of kdump_bundler.sh attached to this KB. 
   
   
4. Ensure that the kdump_bundler script has executable permissions:
   

   chmod +x kdump_bundler.sh

    

5. Execute the script with the corresponding timestamp argument. Any packages downloaded will be available based on the timestamp provided.
   

   ./kdump_bundler.sh -t <timestamp>

   • The timestamp format is YYYYMMDDTHHmmSSZ
     
     
   • For VKR v1.35.2, use timestamp: 20260311T000000Z
     
     
     
6. Confirm that a directory and a "kdump_snapshot_<timestamp>.tar.gz" file with deb files were created.
   
   
   
7. Send the kdump_snapshot_<timestamp>.tar.gz file to your local machine to be sent to the airgapped node.

 

Enable Kernel Dump - Airgapped Node

1. Ensure that the airgapped node is on the same Ubuntu OS version as the previous steps.
   If not the same version, the previous steps will need to be redone with a matching version.
   
   
2. Transfer the kdump_snapshot_<timestamp>.tar.gz file to the airgapped node.
   
   
3. Extract the kdump_snapshot_<timestamp>.tar.gz file into a new directory "kdump_install":
   

   mkdir -p ~/kdump_install
   tar -xvzf kdump_bundle_*.tar.gz -C ~/kdump_install	

    

4. Install the extracted packages:
   

   cd ~/kdump_install
   sudo DEBIAN_FRONTEND=noninteractive apt install -y ./*.deb

    

5. Confirm kdump is working:
   

   echo 'USE_KDUMP=1' | sudo tee /etc/default/kdump-tools

    

6. Configure Kernel Memory:
   

   sudo bash -c "cat << 'EOF' > /etc/default/grub.d/99-zz-kdump-manual.cfg
   # We append to the existing variable so we don't break previous settings
   GRUB_CMDLINE_LINUX_DEFAULT=\"\${GRUB_CMDLINE_LINUX_DEFAULT} crashkernel=2G-:256M panic_on_warn=0 softlockup_panic=1 nmi_watchdog=panic,1\"
   EOF"

    

   • Explanation of parameters:

     • softlockup_panic=1 and nmi_watchdog=panic,1 are policy choices that make a specific failure mode (single-CPU kernel runaway) diagnosable instead of leading to a silent hang. They are recommended, but not strictly required.
       
       
     • crashkernel allocates memory to be reserved for kernel crashes. In the example above, for any VM setting with 2G and above RAM allocation, 256M will be reserved for kernel crash dump collection. Adjust this value based on the size of the VM.
       
       
       
7. Run updates to the grub and reboot the node:
   

   sudo update-grub
   sudo reboot