Virtual Machine unresponsive and snapshot consolidation fails with the error "An error occurred while consolidating disks: 110 (Connection timed out)"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• After deleting a snapshot on a large virtual machine (e.g., Oracle DB), the virtual machine comes up with a warning stating that it requires consolidation.
  
  
• When attempting to consolidate, the consolidation fails with errors such as GenericVmConfigFault and An error occurred while consolidating disks: 110 (Connection timed out). 
  
  
• This specific failure is reflected in the virtual machine's task history within vCenter:
  
  
  
  
• Additionally, the event is captured in the vCenter /var/log/vmware/vpxd/vpxd.log file:
  --> Result:
--> (vim.fault.GenericVmConfigFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.snapshot.vigor.consolidate.error",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "110 (Connection timed out)"
-->             }
-->          ],
-->          message = "An error occurred while consolidating disks: 110 (Connection timed out)."
-->       }
-->    ],
-->    reason = "An error occurred while consolidating disks: 110 (Connection timed out)."
-->    msg = "An error occurred while consolidating disks: 110 (Connection timed out)."
--> }
--> Args:
-->
[timestamp] info vpxd[06333] [Originator@6876 sub=cdrsPlmt opID=CdrsLoadBalancer-####################] Load-balance iteration 5525 scanned 9086120 vmotion 5 min 29 max 4663118 total 869590269 hosts 0 skipped 0 ######-###: false

 

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

• This issue typically arises when a snapshot was taken while the datastore was full and then manually removed after space was extended on the datastore.
• Residual Changed Block Tracking (CBT) metadata within the virtual machine configuration and disk descriptor files can interfere with disk consolidation.
• The default timeout may be insufficient when dealing with large multi-disk virtual machines.

Resolution

1. Remove CBT Metadata References:
Comment out changeTrackPath in the .vmdk and .vmx files: 

sed -i 's/^changeTrackPath/#changeTrackPath/' <vmdk file>

Reference: VM power-on fails with error: Change tracking invalid or disk in use

2. Refresh Metadata:
Take a new snapshot and immediately remove it. This refreshes internal metadata without reintroducing prior corruption.

3. Verify Snapshot Chain:
Check each VMDK in the virtual machine's disk chain for integrity:

vmkfstools -q -v1 <vmdk file>

Reference: Understanding the relationship between Snapshot Manager and the backing files in vSphere 5.x

4. Increase Consolidation Timeout and then retry the Consolidation:

Add the following setting to the virtual machine's .vmx file or via advanced configuration:

diskLib.consolidateTimeOut = "30"

Reference: Increasing the time limit on snapshot consolidation

Initiate the consolidation process again. Due to the virtual machine's size and number of disks, this operation may take an extended amount of time.

(or)

Initiate the consolidation task directly from the ESXi host to bypasses the vpxa-to-vCenter communication channel entirely.
This allows the host's native hostd process to complete the consolidation without having to satisfy vCenter's strict management timeout requirements.

Next Steps:

1. If consolidation fails again, retry consolidation on the affected disk(s).

2. Repeat until all disks are successfully consolidated.

3. Once completed, power on the virtual machine and verify normal operation.

4. If none of the above options work, clone the virtual machine.

Additional Information

• This issue is commonly seen with large virtual machines that have 10 or more disks and where consolidation runs beyond the default timeout threshold.

• CBT references may persist even after snapshot removal, especially when storage pressure or metadata corruption occurred.