vMotion fails with Status - Failed waiting for data. Error 195887124. Out of memory Heap portsetHeap already at its maximum size.
book
Article ID: 317917
calendar_today
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
vMotion of VMs in powered on state files with:
Failed waiting for data. Error 195887124. Out of memory Heap portsetHeap already at its maximum size.
vmkernel.log contains the following messages:
2021-04-02T15:06:16.097Z cpu1:6872978)NetPortset: 2954: Failed to alloc port by portID 0x800036a
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: NetDVS: 4489: failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 139. Failed with error status: Out of memory
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 3656: 3929437244106604596 D: failed to get DVS state in the restore phase from the source host <192.168.192.121>
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 5954: 3929437244106604596 D: Failed handling message reply GET_DVS_STATE: Out of memory
the vmware.log of the virtual machines has the following entries:
2021-04-02T15:06:16.105Z| vmx| I125: Migrate: Caching migration error message list:
2021-04-02T15:06:16.105Z| vmx| I125: [msg.migrate.waitdata.platform] Failed waiting for data. Error bad0014. Out of memory.
2021-04-02T15:06:16.105Z| vmx| I125: [vob.vmotion.dvs.state.restore.failed] vMotion migration [c0a8c079:3929437244106604596] failed to get DVS state in the restore phase from the source host <192.168.192.121>
2021-04-02T15:06:16.105Z| vmx| I125: [vob.net.dvport.reserve.failure] Failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 139. Failed with error status: Out of memory.
2021-04-02T15:06:16.105Z| vmx| I125: [vob.heap.grow.max.reached] Heap portsetHeap already at its maximum size of 374526376. Cannot expand.
2021-04-02T15:06:16.105Z| vmx| I125: Migrate: cleaning up migration state.
2021-04-02T15:06:16.105Z| vmx| I125: Migrate: Final status reported through Vigor.
2021-04-02T15:06:16.105Z| vmx| I125: MigrateSetState: Transitioning from state 12 to 0.
2021-04-02T15:06:16.105Z| vmx| I125: Module 'Migrate' power on failed.
vmkernel.log on the destination host contains these messages:
2021-04-02T15:06:16.096Z cpu2:6872979)Migrate: 103: 3929437244106604596 D: MigrateState: Precopying memory
2021-04-02T15:06:16.097Z cpu1:6872978)NetPortset: 2954: Failed to alloc port by portID 0x800036a
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: NetDVS: 4489: failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 139. Failed with error status: Out of memory
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 3656: 3929437244106604596 D: failed to get DVS state in the restore phase from the source host <192.168.192.121>
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: VMotionSend: 5954: 3929437244106604596 D: Failed handling message reply GET_DVS_STATE: Out of memory
2021-04-02T15:06:16.097Z cpu1:6872978)Migrate: 103: 3929437244106604596 D: MigrateState: Failed
2021-04-02T15:06:16.097Z cpu1:6872978)WARNING: Migrate: 254: 3929437244106604596 D: Failed: Out of memory (0xbad0014) @0x42003a8a8f02
2021-04-02T15:06:16.104Z cpu12:6872961)WARNING: Migrate: 6248: 3929437244106604596 D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
Environment
VMware vSphere ESXi 7.0.x
Cause
This issue occurs due to memory leaks for pktlist in portset world data. When there is no packets attached to pktlist in portset world data, there is a risk to leak the pktlist memory. If the portset world data is not released in the fastslab, then it will be no memory leak as the packet list can be reused. But if the fastslab release this slab to buffer, then the packet list will be leaked.