vMotion fails with DVFilter vMotion error: Failed: Out of memory (0xbad0014) or Failure (0xbad0001)

Products

• VMware vSphere ESXi
• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

vMotion migration fails during the memory precopy phase with one or more of the following errors in the vmkernel.log or vmware.log:

• Heap dvfilterVMotion may only grow by #### bytes (####/####), which is not enough for allocation of #### bytes
• vMotion migration failed to get DVFilter state from the source host
• Failed waiting for data. Error 195887124. Out of memory.
• WARNING: VMotionRecv: ####: DVFilter vMotion error

 

Environment

• VMware vSphere ESXi
• VMware NSX
• VMware vDefend Firewall

Cause

The dvfilterVMotion heap is used to allocate space for temporary data when transferring the Distributed Firewall (DFW) state during migration. If the DFW state is large (due to many rules or complex configurations), the default maximum heap size (~512MB) may be exhausted, causing the vMotion to fail.

Resolution

Choose one of the following methods to resolve the issue:

Method 1: Immediate Workaround (No Reboot Required) This method allows for the migration of the VM so the host can be placed into maintenance mode:

1. Log in to the NSX Manager.
2. Navigate to Security > Distributed Firewall > Settings.
3. Select User Excluded Groups and click Manage Exclusion List.
4. Add the affected Virtual Machine to the Exclusion List.
   • Note: This removes DFW protection from the VM. Ensure appropriate security measures are in place.
5. Perform the vMotion migration.
6. Once the migration is complete and the host is serviced, remove the VM from the exclusion list.

 

Method 2: Permanent Fix (Requires Host Reboot) Increase the maximum heap size for the dvfilter module.

1. SSH to the affected ESXi host as root.
2. Increase the heap limit by running: esxcfg-module -s DVFILTER_HEAP_MAX_SIZE=536875432 dvfilter
3. Reboot the ESXi host for the changes to take effect.
4. Verify the new limit using vsish: get /system/heaps/dvfilterVMotion-####/stats

Additional Information

To check current heap usage without a reboot:

1. Run vsish.
2. Navigate to cd /system/heaps/.
3. Locate the dvfilterVMotion entry and run get stats
4. Example output:

/system/heaps/dvfilterVMotion-0x43144da00000/> get stats
Heap stats {
   Name:dvfilterVMotion
   owning module id:0
...
   current heap size:111016
...
   maximum heap size:536875432