vMotion fails with Status - Failed waiting for data. Error 195887124. Out of memory Heap portsetHeap already at its maximum size.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vMotion of VMs in powered on state files with:

Failed waiting for data. Error 195887124. Out of memory Heap portsetHeap already at its maximum size.

• vmkernel.log contains the following messages:

  2021-04-02T15:06:16.097Z cpu1:6872978)NetPortset: 2954: Failed to alloc port by portID 0x800036a
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: NetDVS: 4489: failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 139. Failed with error status: Out of memory
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 3656: 3929437244106604596 D: failed to get DVS state in the restore phase from the source host <192.168.192.121>
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 5954: 3929437244106604596 D: Failed handling message reply GET_DVS_STATE: Out of memory

• the vmware.log of the virtual machines has the following entries:

  2021-04-02T15:06:16.105Z| vmx| I125: Migrate: Caching migration error message list:
  2021-04-02T15:06:16.105Z| vmx| I125: [msg.migrate.waitdata.platform] Failed waiting for data.  Error bad0014. Out of memory.
  2021-04-02T15:06:16.105Z| vmx| I125: [vob.vmotion.dvs.state.restore.failed] vMotion migration [c0a8c079:3929437244106604596] failed to get DVS state in the restore phase from the source host <192.168.192.121>
  2021-04-02T15:06:16.105Z| vmx| I125: [vob.net.dvport.reserve.failure] Failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 139. Failed with error status: Out of memory.
  2021-04-02T15:06:16.105Z| vmx| I125: [vob.heap.grow.max.reached] Heap portsetHeap already at its maximum size of 374526376. Cannot expand.
  2021-04-02T15:06:16.105Z| vmx| I125: Migrate: cleaning up migration state.
  2021-04-02T15:06:16.105Z| vmx| I125: Migrate: Final status reported through Vigor.
  2021-04-02T15:06:16.105Z| vmx| I125: MigrateSetState: Transitioning from state 12 to 0.
  2021-04-02T15:06:16.105Z| vmx| I125: Module 'Migrate' power on failed.

 

• vmkernel.log on the destination host contains these messages:

  2021-04-02T15:06:16.096Z cpu2:6872979)Migrate: 103: 3929437244106604596 D: MigrateState: Precopying memory
  2021-04-02T15:06:16.097Z cpu1:6872978)NetPortset: 2954: Failed to alloc port by portID 0x800036a
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: NetDVS: 4489: failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 139. Failed with error status: Out of memory
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 3656: 3929437244106604596 D: failed to get DVS state in the restore phase from the source host <192.168.192.121>
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 5954: 3929437244106604596 D: Failed handling message reply GET_DVS_STATE: Out of memory
  2021-04-02T15:06:16.097Z cpu1:6872978)Migrate: 103: 3929437244106604596 D: MigrateState: Failed
  2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: Migrate: 254: 3929437244106604596 D: Failed: Out of memory (0xbad0014) @0x42003a8a8f02
  2021-04-02T15:06:16.104Z cpu12:6872961)WARNING: Migrate: 6248: 3929437244106604596 D: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
  

Environment

VMware vSphere ESXi 7.0.x

Cause

This issue occurs due to memory leaks for pktlist in portset world data.  When there is no packets attached to pktlist in portset world data, there is a risk to leak the pktlist memory.
If the portset world data is not released in the fastslab, then it will be no memory leak as the packet list can be reused. But if the fastslab release this slab to buffer, then the packet list will be leaked.

Resolution

This issue has been resolved in VMware ESXi 7.0 Update 2c

Workaround:
In case an update to ESXi 7.0 U2c or newer is currently not possible, As a workaround use one of the following options:

1. Power off the VM and perform cold migration.
2. Reboot the host.