How to collect memory dump for Supervisor to investigate the Kernel Panic

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

This article provides instructions on how to configure a Supervisor VM and capture a memory dump when a kernel panic is reproducible, ensuring adequate data is collected for root cause analysis (RCA).

If you need to apply the procedure described in this KB to your environment, open a support request with Broadcom Support.

Environment

vCenter Server (Supervisor)

Resolution

Step 1. Configure the Kernel Parameter on the Supervisor VM

Before the issue recurs, disable the automatic reboot on panic to ensure the VM stays in a panicked state for memory capture.

ssh VCENTER
# Check the supervisor password
/usr/lib/vmware-wcp/decryptK8Pwd.py
ssh root@<SUPERVISOR_IPADDR>

# Set - No auto-reboot after panic
sysctl -w kernel.panic=0

 

Step 2. Capture Data and Recover the VM After a Kernel Panic

When the Supervisor VM experiences a kernel panic, perform the following steps immediately before restarting the VM.

2-1. Generate a Memory Snapshot

1. Log in to the ESXi Host Client of the specific host where the panicked Supervisor VM is currently registered.
2. Right-click the affected Supervisor VM and select Take Snapshot.

Important: Ensure that the Snapshot the virtual machine's memory checkbox is explicitly selected before confirming.

2-2. Export the Log Bundle via vCenter Server

1. Log in to the vSphere Client.
2. Right-click the host or cluster and navigate to Export System Logs.
3. In the log selection wizard, verify that VirtualMachines --> SnapshotsAndMemoryImages is checked.
4. Click Export Logs to download the bundle containing the memory state.

2-3. Restart the Supervisor VM

1. From the ESXi Host Client, perform a hard Power Off on the Supervisor VM.
2. Once the VM enters a powered-off state, ESX Agent Manager (EAM) will automatically detect it and power the Supervisor VM back on.

 

Step 3. Revert Configurations and Clean Up

# Log back in to the Supervisor VM via SSH.
# Revert the panic timeout to its default value (10 seconds) by running:
sysctl -w kernel.panic=10

Delete the Temporary Snapshot.

1. Log in to the ESXi Host Client.
2. Open the snapshot manager for the Supervisor VM and delete the memory snapshot created in Step 2 to prevent any performance overhead.

After completing the procedure, upload the ESXi log bundle to the support case.