Long stun time while backing up virtual machine located in vVOL Datastores.

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Snapshot consolidation on vVOL-based virtual machines causes the virtual machines to be stunned while the snapshot vVOL is being deleted.

• In vmware.log of the virtual machine which has the impact of high stun time, you may see similar error:

Checkpoint_Unstun: vm stopped for 63104614 us    (63.1 seconds of stun time) 

• In vvold.log, you may see similar error:

error vvold[721DB70] [Originator@xxxx sub=Default]

--> VasaOp::ThrowFromSessionError [#171241]: ===> FINAL FAILURE deleteVirtualVolume, error (RESOURCE_BUSY / Vvol with id rfc4xxxx.26f5a077-xxxx-xxxx-b5f6-6yx5xxyyyexy is still bound / ) VP (Storage_Group) Container (0x00000x0000xxxxxxxxxxxxxxxxxxxx) timeElapsed=98 msecs (#outstanding 0)

• In vmware.log, you may see a similar error:

vmx| I125: SnapshotVMX_Consolidate: Starting online snapshot consolidate operation.

rfc4xxxx.26f5a077-xxxx-xxxx-b5f6-6yx5xxyyyexy pathToDevice:/vmfs/devices/vvol/0x00000x-0000-xxxx-xxxxxxxxxxxxxxx' : Could not find the file 393219 CID:rfc4xxxx.26f5a077-xxxx-xxxx-b5f6-6yx5xxyyyexyvcpu-0| I125: 2429318:VVOLLIB : VVolLibUnbind:8028: Can't open '

vcpu-0| I125: 2429318:VVOLLIB : VVolLibConvertSoapFault:1406: client.VvolLibDoRetryCheckAndUpdateError failed with a fault

vcpu-0| I125: 2429318:VVOLLIB : VVolLibConvertVvolStorageFault:902: Storage Fault RESOURCE_BUSY (14): Vvol with id rfc4xxxx.26f5a077-xxxx-xxxx-b5f6-6yx5xxyyyexy is still bound /

vcpu-0| I125: 2429318:VVOLLIB : VVolLib_DeleteVVol:3880: VVol: rfc4xxxx.26f5a077-xxxx-xxxx-b5f6-6yx5xxyyyexy CID: 0x00000x0000xxxxx-xxxxxxxxxxxxxxx, delete VASA call failed with ResourceInUse fault, retrying (time elapsed xx secs)...

 

 

Environment

VMware vCenter Server Appliance 6.7.x
VMware vSphere ESXi 6.5
VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vSphere ESXi 6.7
VMware vCenter Server Appliance 6.5.x

Cause

This issue occurs as the snapshot vVOL could not be deleted due to a bind operation on the vVOL held by another ESXi host.


 

Resolution

To resolve this issue place the virtual machine which exists in the vVOL datastore and the virtual machine which involves in the backup process on the same ESXi host. This will ensure that the bind operation is held by the same ESXi host and snapshot consolidation completes without causing long pause.

Note :
The bind on the snapshot vVOL is found to be held by the ESXi host hosting the Backup (proxy) virtual machine.
The vVOL delete succeeds, as soon as the bind is released,  the VM is un-stunned and become accessible again. 

Additional Information

Impact/Risks:
No impact / risks