Out of memory errors on ESXi host with 128 or more logical CPU's and 24 or more vSAN disks.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

Out of memory errors will occur when trying to allocate from any vmkernel heap due to extreme usage of the vmkernel heapMgr's VA space by vSAN IORETRY lfHelper worlds.

You see the following messages in vmkernel.log:

2020-08-01T12:05:44.910Z cpu143:2465355)LSOMCommon: IORETRY_Create:3674: An IORETRY queue for diskUUID 521b2ae4-c2db-a9e7-7250-axxxxxxxxxx6 (0x45018f826950) is encrypted
2020-08-01T12:05:44.938Z cpu143:2465355)HeapMgr: 820: Failed to allocate VASpace for heap. HeapMgr's VASpace exhausted.
2020-08-01T12:05:44.938Z cpu143:2465355)WARNING: Heap: 1099: Could not allocate 8192 bytes for heap worldGroup.4707130: status=Insufficient address space for operation
2020-08-01T12:05:44.942Z cpu143:2465355)WARNING: LSOMCommon: IORETRY_LFHelperInit:206: Unable to create LSOM LFHelper: Out of memory
2020-08-01T12:05:44.942Z cpu143:2465355)LSOMCommon: IORETRY_Create:3691: Allocating IORETRY_EgressLFHelper failed with status Out of memory
2020-08-01T12:05:44.942Z cpu143:2465355)LSOMCommon: IORETRY_Create:3810: Failed creating an IORETRY handle with status Out of memory
2020-08-01T12:05:44.955Z cpu143:2465355)WARNING: PLOG: PLOG_OpenDevHandles:2489: Unable to create IOretry handle for device 521b2ae4-c2db-a9e7-7250-a41d53161fd6 : Out of memory
2020-08-01T12:05:44.955Z cpu143:2465355)WARNING: PLOG: PLOG_InitMDDevice:2602: Unable to create device handles for device 521b2ae4-c2db-a9e7-7250-a41d53161fd6 : Out of memory
2020-08-01T12:05:44.955Z cpu143:2465355)WARNING: PLOG: PLOGOpenDevice:4478: Failed to initialize MD (naa.5000c500af28ba3f:2) with UUID 521b2ae4-c2db-a9e7-7250-a41d53161fd6: Out of memory

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

ESXi version 7.0U1

ESXi host with 128 or more logical CPU's and 24 or more vSAN disks

Cause

During vSAN device discovery, one lfHelper world causing the memory issue.

Resolution

The issue is fixed in vCenter and ESXi version 7.0U2 and later.

If need any further assistance on this, please contact Broadcom Support.

Additional Information

Impact/Risks:

Disks fail to mount due to out of memory as the vmkernel heap VA space is exhausted.