VMs with NVMe Controllers enabled become unresponsive after upgrading to vSphere 8.0 P05

Products

• VMware vSphere ESXi
• VMware vSphere ESX 8.x

Issue/Introduction

 

• The virtual machines on the hosts become unresponsive to ping, SSH, or RDP requests. They remain in hung state until the host is rebooted. Although they do not disconnect, they stay non-responsive until the ESXi reboot is performed.
  
  
• vmkernel log (/var/run/log/vmkernel.log) may report out of memory errors as below
  YYYY-MM-DDTHH:MM:SS vmkernel: cpu##:#####)VNVME: ###: Error status: Out of memory converted to: 0x0:0x#
YYYY-MM-DDTHH:MM:SS vmkernel: cpu##:#####)VNVME: ###: Error status: Out of memory converted to: 0x0:0x#
  
  
• vmware.log (/vmfs/volumes/datastore/vmname/) may report below pattern of READ & WRITE failures,
  
  vvol:#####/rfc#####/vmware-#####.log:YYYY-MM-DDTHH:MM:SS In(##) vcpu-# - NVME-VMK: nvme0:0: WRITE Command failed. Status: 0x0/0x4.
vvol:#####/rfc#####/vmware-#####.log:YYYY-MM-DDTHH:MM:SS In(##) vcpu-# - NVME-VMK: nvme0:0: READ  Command failed. Status: 0x0/0x4
  
  

Environment

vSphere 8.x

Cause

Memory leak has been observed when executing NVMe UNMAP commands on virtual machines with hardware versions earlier than 19. This behavior occurs because NVMe native emulation is not supported on VM hardware versions below 19, causing the commands to be handled differently.

Resolution

Upgrade the VM hardware version to 19 or later.

If an upgrade is not possible, proceed with disabling unmap for the Virtual Machine.

1. Login vSphere Client
2. Power off the VM
3. Edit the VM settings in the vCenter UI
4. Go to Advanced Parameters tab and filter for disk.scsiUnmapAllowed
   • If present but set to True, change the value to False

     
     
   • If not present add the attribute disk.scsiUnmapAllowed and set value to False

     
     
     
     
5. Save the configuration with the OK button
   
   
6. Power on the VM