Longer VM stun times and snapshot create/delete operations after upgrading to ESXi 6.7 P06 with large VMFS datastores

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
After upgrading to ESXi 6.7 P06 (Build # 18828794), longer stun times and snapshot create/delete operations can occur on larger VMFS datastores when using Change Block Tracking (CBT). These long stun times lead to applications in VMs being unresponsive or not reachable on the network or transaction timeouts or any similar disruption.

Environment

VMware vSphere ESXi 6.7
VMware ESXi 6.7.x

Cause

The long VM stun time reported during snapshot create was due to the time taken to search for suitable Resource Clusters (RC) to affinitize allocations to .CTK files used for CBT. This issue is exasperated on large datastores (32TB or greater).

Resolution

This issue will be addressed in ESXi 6.7 P07. The issue is also already solved in ESXi 7.0 P04 due to a code logic change in that release.

Workaround:

While utilizing much smaller datastores will help with this issue, the only workaround is to enable the following advanced configuration parameter, which is not persistent across reboots:

vsish -e set /config/VMFS3/intOpts/RCLockWaitThresholdVMFS6 0