Multiple VM's fails during snapshot consolidation with error "Bad file descriptor"

Products

• VMware vSphere ESX 8.x

Issue/Introduction

- Multiple VMs fails during consolidation which leads VM to be in hung state with power-on/off options greyed out.

- Attempts to consolidate disks on a VM fails with below error:
Task Details:
Name: consolidateDisks
An error occurred while consolidating disks: The operation failed. Consolidation failed for disk node 'scsi0:0*: The operation failed. Failed to send allocated bitmap to VMkernel while copying disks from /vmfs/ volumes/6882xxxx-a3fxxxxx-82xx-e411xxxxxx/VM-name/VM.vmdk' to '/vmfs/ volumes/6882xxxx-a3fxxxxx-82xx-e411xxxxxx/VM-name/VM_1.vmdk'

- In /var/log/hostd.log, we see below error with consolidation of delta disks fails 

[YYYY-MM-DDTHH:MM:SS] In(05) worker-8311338 - OBJLIB-FILEBE : FileBEIoctl: ioctl operation IOCTLCMD_VMFS_DELTADISKS(3033) failed on '/vmfs/devices/deltadisks/30bc226d-xxxxxx_3-000019-sesparse.vmdk' : Bad file descriptor (589826)
[YYYY-MM-DDTHH:MM:SS] In(05) worker-8311338 - DISKLIB-VMFS :VmfsSparseExtentCommonGetAllocatedSectorChunks: ObjLib_Ioctl failed 0x90009
[YYYY-MM-DDTHH:MM:SS] Er(02) worker-8311338 - DISKLIB-LINK : DiskLinkGetAllocatedSectorChunks: Failed to get allocated sector chunks: Bad file descriptor
[YYYY-MM-DDTHH:MM:SS] Er(02) worker-8311338 - DISKLIB-LIB_MISC : DiskLibGetAllocatedSectorChunksInRangeInt: failed to get allocated sector bitmap: Bad file descriptor
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-8311338 - Mirror: scsi0:0: Failed to get allocated sectors: Bad file descriptor.
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-8311338 - Mirror: scsi0:0: Mirror_SendBitmapToKernel: Failed to send allocated bitmap to VMkernel
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - MirrorDiskCopyGetCopyProgress: Bitmap copy failed on disk /vmfs/volumes/xxxxxx-e4115b9849ea/xxxx/xxxx-000019.vmdk.

Environment

vSphere 8.x

Cause

The issue occurs due to corruption in sespare file possibly due to storage array issue. 

Resolution

It is recommended to review and address the storage performance issues, take the necessary steps to improve overall performance, and then attempt the VM consolidation again.

For further assistance, you can open a support case with Broadcom by selecting ESXi Storage: Storage as the product through the Broadcom Support Portal

Additional Information

Failed to power on virtual machine

Impact/Risks:
Ensure you have good backup of your virtual machines. Some recovery procedures can risk making the problem worse if not followed correctly.