HP replication plugin causes virtual machine crashes during vMotion or snapshot creation

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Reviewing the vCenter Tasks and Events for the impacted VMs revealed the following behaviors and error logs:
  
  
  • A critical event is logged just before the crash: Error message from [host]: VMware ESX unrecoverable error: (vcpu-0) Non-empty heap (hpereplication) being destroyed (avail is ####, should be ####).
    
    
  • The following error snippet was observed in the /var/run/log/hostd.log of an ESXi host, indicating a failed snapshot creation task for a Zerto-protected virtual machine:
    
    

    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/[Datastore-Name]/[VM-Name]/[VM-Name.vmx] Handling vmx message 6049545: VMware ESX unrecoverable error: (vcpu-0)
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: --> Non-empty heap (hpereplication) being destroyed (avail is ####, should be ####).
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: --> A log file is available in "/vmfs/volumes/[Datastore-Name]/[VM-Name]/vmware.log".
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: --> You can request support.
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: -->
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: --> To collect data to submit to VMware technical support, run "vm-support".
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: --> We will respond on the basis of your support entitlement.
    YYYY-MM-DDTHH:MM:SS Db(167) Hostd[#######]: -->
    YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event ##### : Error message on [VM-Name] on [ESXI Hostname] in ha-datacenter: We will respond on the basis of your support entitlement.

• This can occur in Zerto protected virtual machines as well where sudden crashes and connectivity losses during attempted vMotion (hot migration) or non-memory snapshot operations.

Environment

vCenter Server 8.x
vSphere ESXi 8.x

Cause

The core issue preventing successful snapshots and vMotions is the hpereplication IOfilter attached to the virtual machine's disks.

When a snapshot or vMotion is initiated, the hpereplication IOfilter encounters an unrecoverable heap error, which forcefully crashes the virtual machine.

Resolution

Reach out to HPE to investigate why the VMX crash occurred. 

Workaround:

To isolate the IOfilter as the source of the failure and permit standard vMotion or snapshot operations, the IOfilter must be bypassed. Proceed with the following steps:

1. Power-off the Virtual Machine.
   
   
2. Locate the .vmdk descriptor file for the affected virtual machine.
   
   
3. Open the .vmdk file and locate the ddb.sidecars entry.
   
   
4. Comment out the line by adding a # at the beginning.
   • Example: # ddb.sidecars = "hpereplication_[X],[VM_NAME]_[ID].vmfd;hpereplication_[Y],[VM_NAME]_[ID].vmfd"
     
     
5. Perform the vMotion or snapshot operation again to verify success.

If the identified IOfilter is associated with Zerto, please engage Zerto support for further troubleshooting.