ESXi Installation or VMFS creation on Micron 7500 or Solidigm P5336 NVMe drives may cause PSOD

Products

• VMware vSphere ESX 8.x

Issue/Introduction

• Fresh installation or creating VMFS partitions on a Micron 7500 or Solidigm P5336 NVMe drives might fail with a purple diagnostic screen with an error such as Exception 14 or Corruption in dlmalloc.
  
  
• The PSOD stack might vary as the issue causes heap corruption, following are two sample PSOD stacks :
  
  
  
  

Environment

vSphere ESXi 8.0 U2
vSphere ESXi 8.0 U3

Cause

UNMAP commands enable ESXi hosts to release storage space that is mapped to data deleted from the host. In NVMe, the equivalent of UNMAP commands is a deallocate DSM request. Micron 7500 and Solidigm P5336 devices advertise a very large value in one of the deallocate limit attributes, DMSRL, which is the maximum number of logical blocks in a single range for a Dataset Management command. This leads to an integer overflow when the ESXi unmap split code converts number of blocks to number of bytes, which in turn might cause a failure of either installation or VMFS creation.

Resolution

This is a known issue with vSphere ESXi 8.0 U2 and above versions, engineering is actively working towards fixing this issue in a future release.

Workaround

Perform below steps to workaround the issue based on the use-case:

Fresh ESXi Installation:

• Install ESXi 8.0 GA (or any build prior to 8.0 U2).
• Login to ESXi host after installation using Shell or SSH.
• Deactivate UNMAP for the affected disk by using the below command.
  
  esxcli storage core device vaai status set -D 0 -d <device-id> 
  
  
• Upgrade ESXi host to 8.0 U2 or U3 (depending on the requirement).

Creating VMFS Partitions:

• Login to ESXi host after installation using Shell or SSH.
• Deactivate UNMAP for the affected disk by using the below command.
  
  esxcli storage core device vaai status set -D 0 -d <device-id>
  
  
• Create the VMFS partition.
  
  Note: UNMAP can be reactivated after creating the VMFS datastore, but UNMAP must remain deactivated before deleting or creating the partition.