vCenter Server Appliance crashes with "Kernel panic - not syncing"

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter Server Appliance (vCSA) may experience sudden, critical crashes resulting in a completely unresponsive state. 
• During these events, the VM console displays a "Kernel panic - not syncing" message similar to:
  
  

  [2758060.200949] RIP: 0010: exc_page_fault+0x1c/0x130
[2758060.200950] Code: CC CC CC CC CC CC CC CC CC cc cc cc cc cc 55 48 89 e5 41
57 41 56 41 55 49 89 f5 41 54 49 89 fc 0f 20 d0 0f 1f 40 00 49 89 c7 <65> 48 8b
04 25 c0 fb 01 00 48 8b 80 08 08 00 00 0f 0d 48 78 0f 1f
[2758060.200950] RSP: 0018:fffffe7ea16b7008 EFLAGS: 00010083
[2758060.200950] RAX: ffff95d395e9fbc0 RBX: 0000000000000000 RCX: ffffffffadc011
27
[2758060.200951] RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffe7ea16b70
38
[2758060.200951] RBP: fffffe7ea16b7028 R08: 0000000000000000 R09: 00000000000000
00
[2758060.200951] R10: 0000000000000000 R11: 0000000000000000 R12: fffffe7ea16b70
38
[2758060.200951] R13: 0000000000000000 R14: 0000000000000000 R15: ffff95d395e9fb
c0
[2758060.200951] FS:
0000000000000
[2758060.200951] CS:
[2758060.200951] CR2: ffffffff8d8011b6 CR3: 00000002ac73a003 CR4: 0000000000770e
e0
[2758060.200951] PKRU: 55555554
[2758060.200952] Kernel panic - not syncing: Fatal exception in interrupt
[2758060.200994] Kernel Offset: 0x2c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff )

• In /vmfs/volumes/Datastore/VMFolder/vmware.log, you see entries similar to:
  
  YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <6>[2747339.770866] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 2.717 msecs
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <6>[2747339.771953] perf: interrupt took too long (16670 > 9772), lowering kernel.perf_event_max_sample_rate to 11750
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <4>[2758060.109183] WARNING: stack recursion on stack type 5
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <0>[2758060.109233] traps: PANIC: double fault, error_code: 0x0
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <4>[2758060.109234] double fault: 0000 [#1] SMP NOPTI
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <4>[2758060.109234] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.10.238-2.ph4 #1-photon
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <4>[2758060.109235] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS #.#.# MM/DD/YYYY
YYYY-MM-DDTHH:MM:SS In(05) vcpu-2 - Guest: <4>[2758060.109235] RIP: 0010:exc_page_fault+0x1c/0x130

Environment

VMware vCenter Server 8.x

Cause

The crash is caused by a double fault resulting from stack recursion. Third-party monitoring agents installed on the vCenter appliance interfere with the kernel's Non-Maskable Interrupt (NMI) handling. Specifically, the perf_event_nmi_handler takes too long to execute, leading to interrupt delays and eventual system instability.

Resolution

Broadcom does not support the installation of third-party software, agents, or drivers directly on the vCenter Server Appliance. These modifications can interfere with the appliance's resource management and kernel stability. For more information, please refer Broadcom KB 330057. 

To resolve this issue, the unsupported third-party agents must be disabled and removed from the vCenter Server Appliance.