VMware ESXi snapshot removal task stuck at 98 percent

Products

• VMware vCenter Server

Issue/Introduction

A virtual machine snapshot removal task remains stuck at 98%. Monitoring the VM directory via watch -d 'ls -luth | grep -E "sesparse|flat|delta"' confirms that .vmdk files are not updating. Attempts to kill the VM process via command line fail with "No such process," despite lsof showing active locks by a specific PID. The VM may appear as "Invalid" if registration is attempted on another host. Key symptoms include:

• vmware.log contains: Caught signal 15 -- tid <THREAD_ID>

• vobd.log contains: [vob.uw.core.dumped] /bin/vmx(<PID>) ... vmx-zdump.000

• vmkernel.log contains: Hbr: 1503: Terminating sync task followed by VSCSI: Destroying Device

Environment

 

• VMware ESXi 8.x

• Snapshot-based backup solution (e.g., Veeam)

• Hypervisor-based replication (HBR) / VMware Cloud Director Availability (VCDA)

 

Cause

A deadlock occurs when a snapshot-based backup consolidation task and a hypervisor-based replication (HBR) sync task execute simultaneously on the same CPU thread. The hypervisor attempts to destroy the virtual SCSI (VSCSI) device while the HBR filter is still active, leading to a forced eviction of the filter, a VMX crash (Signal 15), and subsequent zombie processes holding file locks.

Resolution

 

• Evacuate all other healthy Virtual Machines from the affected ESXi host using vMotion.

• Reboot the ESXi host to clear the zombie processes and release the persistent file locks on the VMDKs.

• Register the VM to a healthy host.

• Perform a Snapshot Consolidation manually to merge the remaining delta disks.

• Review Backup and Replication schedules: Ensure that snapshot-based backup windows and hypervisor-based replication sync cycles do not overlap for large workloads.

 

Additional Information

Investigating Virtual Machine file locks on ESXi Host(s) (Broadcom KB 314365)