ESA Storage Latency and VM Performance Impact Caused by ESXi Host PCPU Hang

Products

• VMware vSphere ESXi

Issue/Introduction

In a vSAN ESA cluster, virtual machines may experience elevated read and write latency, resulting in application performance degradation.

The issue may present with the following symptoms:

• A sudden increase in storage latency across the vSAN environment.
• Virtual machines reporting increased disk response times.
• vMotion operations failing or taking significantly longer than expected.
• vSAN network latency increasing across the cluster.
• No corresponding increase in I/O workload will be seen.
• Esxi may go into PSOD state.

 In vmkernel log below events will be observed

2026-05-18T03:30:30.731Z Wa(180) vmkwarning: cpu125:2097372)WARNING: StorageDeviceIO: 201: Device eui.xxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 99 microseconds to 14241 microseconds.
2026-05-18T03:30:53.021Z Wa(180) vmkwarning: cpu46:2098836)WARNING: StorageDeviceIO: 201: Device eui.xxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 54 microseconds to 13936 microseconds.
2026-05-18T03:31:04.625Z Wa(180) vmkwarning: cpu125:2098832)WARNING: StorageDeviceIO: 201: Device eui.xxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 99 microseconds to 15608 microseconds.

Around the same time memory page retire and PCPU locked up events will be seen
vobd.all:2026-05-18T03:30:17.254Z In(14) vobd[2098148]:  [pageretireCorrelator] 23256094833825us: [vob.pageretire.selectedmpnthreshold.host.exceeded] Number of MPNs selected for retirement is 1
vobd.all:2026-05-18T03:30:17.258Z In(14) vobd[2098148]:  [pageretireCorrelator] 23256094834025us: [vob.pageretire.selectedmpnthreshold.host.exceeded] Number of MPNs selected for retirement is 2
vobd.all:2026-05-18T03:38:53.313Z In(14) vobd[2098148]:  [pageretireCorrelator] 23256610871943us: [vob.pageretire.platform.retire.request] Memory page retirement requested by platform firmware. FRU ID: unknown. Refer to System Hardware Log: B5
vobd.all:2026-05-18T03:38:53.988Z In(14) vobd[2098148]:  [pageretireCorrelator] 23256220068485us: [esx.problem.pageretire.platform.retire.request] Memory page retirement requested by platform firmware. FRU ID: unknown. Refer to System Hardware Log: B5
vobd.all:2026-05-18T03:39:09.590Z In(14) vobd[2098148]:  [pageretireCorrelator] 23256610872698us: [vob.pageretire.selectedmpnthreshold.host.exceeded] Number of MPNs selected for retirement is 4
vmkwarning.all:2026-05-18T03:34:53.751Z Wa(180) vmkwarning: cpu3:3307472)WARNING: Heartbeat: 961: PCPU 5 didn't have a heartbeat for 5 seconds, timeout is 10, 1 IPIs sent; *may* be locked up.
vmkwarning.all:2026-05-18T03:39:48.952Z Wa(180) vmkwarning: cpu14:2099398)WARNING: Heartbeat: 961: PCPU 11 didn't have a heartbeat for 5 seconds, timeout is 10, 1 IPIs sent; *may* be locked up.
vobd.all:2026-05-18T03:39:49.029Z In(14) vobd[2098148]:  [cpuCorrelator] 23256666633804us: [vob.cpu.nmi.ipi2.savebt] NMI IPI: PC 0x420022122739, SP 0x453b8a59a110 (Src 0x1, CPU11)
vmkernel.all:2026-05-18T03:39:48.953Z cpu11:84580416)NMI: 738: NMI IPI: PC 0x420022122739, SP 0x453b8a59a110 (Src 0x1, CPU11)
vmkwarning.all:2026-05-18T03:49:06.049Z Wa(180) vmkwarning: cpu35:186233330 opID=7d247bb6)WARNING: Heartbeat: 961: PCPU 53 didn't have a heartbeat for 5 seconds, timeout is 10, 1 IPIs sent; *may* be locked up.

 

Environment

 vSAN ESA 8.x

Cause

The affected host reported memory page retirement events followed by PCPU heartbeat timeout warnings, indicating potential hardware or CPU lockup conditions. As the PCPUs became unresponsive, I/O processing on the host was delayed, resulting in queued storage operations, elevated device latency, increased vSAN network latency, and VM performance degradation.

Resolution

Please engage the hardware vendor to investigate on host memory page retire and PCPU lock up errors.