High Stun time observed in snapshot consolidation

Products

• VMware Cloud on AWS

Issue/Introduction

• During a snapshot creation or removal operation, virtual machines residing on NFS storage become temporarily unresponsive (30 seconds or even more).
• In some cases the ESXi host itself might stop responding as well at the same time.
• Snapshot removal tasks after snapshot-based backups for VMs located on NFS datastores take a long time to complete.
• /var/run/log/vmkernel.log reports expired locks similar to the following examples:

  <timestamp> In(182) vmkernel: cpu76:112395096 opID=###)NFSLock: 3314: lock .lck-2c01e25a00000000 expired: counter prev 351507596f6abf64-052312cf-ab72-06c7617dc159 : curr 351507596f6abf64-052312cf-ab72-06c7617dc159 (loop count 3) 
  <timestamp> In(182) vmkernel: cpu68:112395096 opID=###)NFSLock: 3314: lock .lck-2a01e25a00000000 expired: counter prev 351509256f6abf64-052312cf-ab72-06c7617dc159 : curr 351509256f6abf64-052312cf-ab72-06c7617dc159 (loop count 3) 
  <timestamp> In(182) vmkernel: cpu117:112395096 opID=###)NFSLock: 3314: lock .lck-1901e25a00000000 expired: counter prev 351510866f6abf64-052312cf-ab72-06c7617dc159 : curr 351510866f6abf64-052312cf-ab72-06c7617dc159 (loop count 3)
  

• Additionally when looking at the log for the specific VM, /vmfs/volumes/<datastore>/<vm_name>/<vm_name>.vmx, large stun time values can be seen, for example:

  <timestamp> In(05) vcpu-0 - CPT: vm was stunned for 877178 us
  <timestamp> In(05) vcpu-0 - CPT: vm was stunned for 40115060 us
  

Environment

• VMware vSphere ESXi 8.0.1
• VMware vSphere ESXi 8.0.2

Cause

When multiple hosts open a vmdk in read-only mode, the resulting lock file is marked as shared. Due to this, when the other hosts are done with the locking, the lock file still remains as shared.

Now when the only host, host1, wants to acquire an exclusive (EXCL) lock on the same vmdk, it has upgrade the existing lock from shared to exclusive. This triggers a liveness check for the lock, which takes 30 seconds to break and acquire the lock.

When this happens during a consolidation, the virtual machine itself cannot proceed its ongoing I/O until the liveness check has finished, and is therefore stunned for these 30 seconds, plus the time it takes to pick up on the I/O queued up prior to the liveness check.

Resolution

This issue is resolved in VMware Cloud on AWS 1.24 and VMware vSphere ESXi 8.0 Update 2b (8.0.2 P03), build 23305546.

 

Workaround:

none

Additional Information

Impact/Risks:

The virtual machine will be inaccessible while it is stunned during the snapshot consolidation.