Disk consolidation failed for multiple VMs across multiple hosts.

Products

• VMware vSphere ESXi

Issue/Introduction

Snapshot consolidation for multiple VMs was failing with the error : 

Failed to copy source (/vmfs/volumes/xxxxxx-xxxxxx-xxxx-xxxxxxxxx/VM_Name/VM_Name.vmdk) to destination (/vmfs/volumes/xxxxxx-xxxxxx-xxxx-xxxxxxxxx/VM_Name/VM_Name_3.vmdk): Timeout. An error occurred while consolidating disks: The operation failed. Consolidation failed for disk node 'scsi0:2': The operation failed. Failed to get copy progress while consolidating disks from '/vmfs/volumes/xxxxxx-xxxxxx-xxxx-xxxxxxxxx/VM_Name/VM_Name_3-000001.vmdk' to '/vmfs/volumes/xxxxxx-xxxxxx-xxxx-xxxxxxxxx/VM_Name/VM_Name_3.vmdk'. 

In the ESXi logs we see the errors similar to :

vmkwarning.log

yyyy-mm-ddT02:31:53.960Z cpu58:2098177)WARNING: iodm: vmk_IodmEvent:194: vmhbaX: FRAME DROP event has been observed 8 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
yyyy-mm-ddT03:00:27.497Z cpu51:2175258)WARNING: iodm: vmk_IodmEvent:194: vmhbaX: FRAME DROP event has been observed 6 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
yyyy-mm-ddT03:01:19.620Z cpu5:2100954)WARNING: iodm: vmk_IodmEvent:194: vmhbaX: FRAME DROP event has been observed 7 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
yyyy-mm-ddT03:02:13.032Z cpu25:2109396)WARNING: iodm: vmk_IodmEvent:194: vmhbaX: FRAME DROP event has been observed 6 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

Environment

VMware ESXi 7.x

VMware ESXi 8.x

Cause

FRAME DROP events suggest a possible connectivity issue or congestion. An erroneous FC Port can also lead to this issue.

Resolution

Engage the fabric vendor to investigate and mitigate the issue.