Not aligned unmap triggered from Guest Operating System (GOS) can cause VM to go unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Performance degraded for one or for a few VMs
• Hosts going unresponsive
• The high number of "write-same" traffic seen on an array side at a given point in time 

 

Environment

VMware vSphere 7.0.x

Cause

• In certain scenarios The VM Guest OS (GOS) initiates unmaps IO that are not aligned or granular to 1MB on the disk that is hosted on the VMFS Datastore. As the blocks cannot be unmapped because of the non-alignment or granularity, VMFS initiates zeroing of the corresponding misaligned or non-1MB ranges. 
• Please note: Any sub-range in the guest issued UNMAP that satisfies the 1MB granularity and alignment conditions will be unmapped. This is as per design.
• Zeroing is done via the Data Mover component in ESXi (DM)  which can issue  "write-same" command to the backend  storage array. High numbers of "write-same" commands  sent to array during a time interval can lead to performance degradation or VM going unresponsive. 
• Write-same will only be seen when unmaps are received on VMFS datastore and not on RDMs.

Resolution

• The   Guest Operating System (GOS)    needs to send 1MB aligned and 1MB granular unmaps.
• This in turn will result in all the blocks getting unmapped and VMFS will in turn send unmaps to backend array and not write-same.
• This solution needs to come from  Guest Operating System (GOS) side.
• From VMware side sending write-same in the event of getting non-aligned unmaps is by design and needed for Thin Provisioned Return Zero (TPRZ) compliance

 

Workaround:

• As a work around user can disable UNMAp from GOS 

To disable guest OS unmap in Windows, please perform the following steps:

• Edit Windows registry:

  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\DisableDeleteNotification

  Change the value FROM 0 to 1

  OR

• Using CLI:

fsutil behavior set DisableDeleteNotify 1

• Reboot the Virtual Machine

Note:  This needs to be done on all VMs

 

Additional Information

Impact/Risks:

• VMs might see Performance deterioration or Experience unresponsiveness if there are too many write same commands to the storage backend, resulting from the misaligned or non-granular UNMAPs from the guest. 

• ESXi host might get in to the Hung state