ESXi host might leak memory during NVMe unmap operations
search cancel

ESXi host might leak memory during NVMe unmap operations

book

Article ID: 435455

calendar_today

Updated On:

Products

VMware vSphere ESXi 8.0

Issue/Introduction

ESXi host might leak memory during NVMe unmap operations.

  • The ESXi host connected with NVME devices
  • If an identify namespace call to a NVMe target fails during a NVMe unmap operation

The memory allocated for the unmap command might not be released.
As a result, the higher memory usage may be observed in the ESXi host.
At high loading situation, it may led the ESXi panic (PSoD).

--vmkernel.log--
2025-05-02T01:30:21.788Z cpu54:##########)NvmeDeviceIO: PsaNvmeDeviceIssueIdentifyNamespace:2635: Identify NS command failed on vvol.############## with H:0x6 D:0x0 P:0x0
2025-05-02T01:30:21.788Z cpu54:##########)WARNING: NvmeFds: PsaNvme_SplitDeleteBlocks:1220: Invalid identify namespace 11 behind vPE vvol.##############

--PSOD Backtrace--
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)Backtrace for current CPU #54, worldID=##########, fp=0x0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x453937119fa0:[0x420035811f2e]PanicvPanicInt@vmkernel#nover+0x2ee stack: 0x45393711a058, 0x2e672e65282064, 0x453937119fa0, 0x420000000001, 0x0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a050:[0x420035812758]Panic_NoSave@vmkernel#nover+0x4d stack: 0x45393711a0b0, 0x45393711a070, 0xfa8f1f6201000000, 0x4200360cd42c, 0x23d
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a0b0:[0x420035812941]Panic_OnAssertAt@vmkernel#nover+0xba stack: 0x23d00000000, 0x4200360cd42c, 0x4200360976bb, 0x4200360aa2c7, 0x4200360b6cbc
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a130:[0x420036010ce5]Int6_UD2Assert@vmkernel#nover+0x35e stack: 0x0, 0x4200360070d6, 0x0, 0x0, 0x8005003f
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a150:[0x4200360070d5]gate_entry@vmkernel#nover+0xb6 stack: 0x8005003f, 0x0, 0x3600, 0x3636, 0x4599bf54c8e0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a210:[0x420036041db0]AsyncPopCallbackFrameInt@vmkernel#nover+0x1e4 stack: 0x0, 0x4599406e3918, 0x45393711a920, 0x420035cf6af6, 0x4309f8a49b60
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a240:[0x420035cf6af5]PsaNVMe_DeleteBlocks@vmkernel#nover+0x836 stack: 0x109, 0x4309f9128fc0, 0x0, 0x100000001, 0x453900000100
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a320:[0x420035cf80d3]PsaNVMe_DevIoctl@vmkernel#nover+0x1218 stack: 0x200, 0x432f75e70870, 0x432f75e71a40, 0x42003583eb88, 0x432f75e6d560
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a3f0:[0x420037e2cadb]NvmeExtentDeleteBlocks@(vvol)#<None>+0xe4 stack: 0x432f75e71a40, 0x420037e36c23, 0x4599baa3e740, 0x80000001, 0x420036041f4e
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a640:[0x420037e4ca72]VVolIoctl@(vvol)#<None>+0x144f stack: 0x42003562f743, 0x42003562fe32, 0x80000002, 0x42004d008367, 0x45393a5a1a33
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a8e0:[0x42003567755a]DevFSFileBlockUnmap@vmkernel#nover+0x213 stack: 0x1, 0x23939c000, 0x2000, 0x20000000200, 0xb400000
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a980:[0x42003565990b]FSSVec_FileBlockUnmap@vmkernel#nover+0x1c stack: 0x800000, 0x80000000357a80f1, 0x800000, 0x11c9ce0, 0x20000000010
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711a990:[0x420035f8a6f9]VSCSI_ExecFSSUnmap@vmkernel#nover+0xc6 stack: 0x800000, 0x11c9ce0, 0x20000000010, 0x0, 0x459940661198
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711aa00:[0x420035f8d61f]VSCSI_FSCommand@vmkernel#nover+0x12a4 stack: 0x40000e1bd1, 0x32e1498, 0xffff810047fd1000, 0x7000430e00000000, 0x0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711aac0:[0x420035faad39]VSCSI_IssueCommandBE@vmkernel#nover+0x62 stack: 0x0, 0x0, 0x430e605aea40, 0x1, 0x45393711ac30
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711ab20:[0x420035fad604]VSCSIExecuteCommandInt@vmkernel#nover+0x9f1 stack: 0x0, 0x4200356301f1, 0x431400000000, 0x4599aa54f1a0, 0x4599b8ba6e80
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711ac40:[0x420035fafd11]VSCSI_VmkExecuteCommand@vmkernel#nover+0xa6 stack: 0x430e605aea40, 0x1, 0x430e604cd9c0, 0x4314d855c640, 0x0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711aca0:[0x420035fa0def]PVSCSIVmkProcessCmd@vmkernel#nover+0x8a0 stack: 0x0, 0x0, 0x45390000000c, 0x430e604f10c0, 0x45393711f000
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711adf0:[0x420035fa1608]PVSCSIVmkProcessRequestRing@vmkernel#nover+0x149 stack: 0x45393711ae6c, 0x420035fa168c, 0x3900020000, 0xa5f923890f2c, 0x2f601ac600000001
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711aeb0:[0x420035fa2d50]PVSCSIProcessRingWork@vmkernel#nover+0x15d stack: 0x296, 0x42003601ffc0, 0xa5f923b2a2c0, 0x8500000000, 0x0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711aed0:[0x42003601ffbf]VMMVMKCall_Call@vmkernel#nover+0x1d4 stack: 0x0, 0x0, 0x0, 0x0, 0x0
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)0x45393711afa0:[0x42003601be6f]VMKVMM_ArchEnterVMKernel@vmkernel#nover+0x188 stack: 0xffff45393a59f100, 0xfc408000, 0x45393711f000, 0xfffffffffc407d58, 0xfffffffffc407dc8
YYYY-MM-DDTHH:MM:SSZ cpu54:##########)VMware ESXi 8.0.x [Releasebuild-######## x86_64]
ASSERT bora/vmkernel/main/async_io.c:573

Environment

VMware ESXi 8.0

Resolution

This issue is resolved in the VMware ESXi 8.0 Update 3h (Build 25067014)

For workaround, reboot the ESXi host to release the memory. 

Additional Information

VMware ESXi 8.0 Update 3h Release Notes
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-80u3h-release-notes.html

Powered by Wolken Software