VM consolidation tasks are stuck, VM's are unresponsive whenever the backups are initiated for VMs

Products

• VMware vSphere ESXi

Issue/Introduction

• Snapshot Consolidations are stuck without any progress.
• Unable to perform any tasks on the VM, VMs are not responsive unless we perform a reboot.
• Vmware.log on the ESXI host reports the following error messages while the backup is initiated:
  
  

  /vmfs/volumes/###################/ExampleVM/vmware.log

  ###-##-##T##:##:##.###Z In(05) vmx - SnapshotVMX_Consolidate: Starting online snapshot consolidate operation.
###-##-##T##:##:##.###Z Wa(03) vcpu-0 - Heap_Align(hpereplication, 440/440 bytes, 8 align) failed. caller: 78ECD4A26C
###-##-##T##:##:##.###Z Er(02) vcpu-0 - zdriverFilterC-#2-_zmp_heap_alloc:191: alloc size=432, zmp_occupied[1]=18441456, /tmp/filter_build_temp/zdriver/zsimple_pool.c:22
###-##-##T##:##:##.###Z Er(02) vcpu-0 - zdriverFilterC-#2-zsimple_pool_init:24: failed to allocate element 4779
###-##-##T##:##:##.###Z In(05) vcpu-0 - zdriverFilterC-#4-zvol_start:202: start vol 1001 type=4 stack attached=0 dirtifyBm=0
###-##-##T##:##:##.###Z In(05) vcpu-0 - zdriverFilterC-#4-zfldb_connect_vols_locked:369: connect volumes failed to find stack with specified parameters
###-##-##T##:##:##.###ZZ In(05) vcpu-0 - zdriverFilterC-#4-_zmod_set_ctrl_vol:155: added control volume
###-##-##T##:##:##.###ZZ In(05) vcpu-0 - WORKER: Creating new group with maxThreads=1 (16)
###-##-##T##:##:##.###ZZ In(05) vcpu-0 - zdriverFilterC-#4-zthread_start:352: thread started
###-##-##T##:##:##.###Z In(05) vcpu-0 - WORKER: Creating new group with maxThreads=1 (16)
###-##-##T##:##:##.###Z In(05) vcpu-0 - zdriverFilterC-#4-filterInit:625: called
###-##-##T##:##:##.###Z Er(02) vcpu-0 - zdriverFilterC-#2-readTweaksFile:141: Failed to open tweak file: /var/run/zfilt_tweaks.txt, errno: 2, as string: No such file or directory

  ###-##-##T##:##:##.###Z Er(02) filtPoll - zdriverFilterC-#2-processSocketMessage:907: got n=-1 when rem=24 isRecv=1
###-##-##T##:##:##.###Z Er(02) filtPoll - zdriverFilterC-#2-processSocketMessage:908: errno=Connection timed out

Environment

• VMware vSphere ESXI 7.x
• VMware vSphere ESXI 8.x

Cause

• Snapshot consolidations can become stuck due to heap allocation failures from a third-party backup filter while they are running.
• Backup driverFilter module while snapshot consolidation was running, mostly referring to a storage/filter driver that tried to allocate memory and failed.

Resolution

• The Backup Vendor needs to be contacted for further assistance.