vSAN API QueryPhysicalVsanDisks() may fail with MemoryError.

Products

• VMware vCenter Server

Issue/Introduction

When querying the vSAN API QueryPhysicalVsanDisks(), the API may intermittently fail due to a memory error.

ESXi hostd, vsanmgmt, and vmkernel logs show the following traces for this issue.

hostd.log:

Hostd[]: [Originator sub=vmomi.soapStub[3] opID=xxxxx sid=xxxxxx SOAP request returned HTTP failure; <SSL(), /vsanperf>, method: queryPhysicalVsanDisks; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
Hostd[]: --> faultCause = (vmodl.MethodFault) null,
Hostd[]: --> faultMessage = <unset>,
Hostd[]: --> reason = "MemoryError()"
Hostd[]: --> msg = "Received SOAP response fault from (, /vsanperf>]: queryPhysicalVsanDisks"

vsanmgmt.log:

vsand[]: Daemon memory stats: eMin=174.080MB, eMinPeak=208.896MB, rMinPeak=209.856MB MEMORY PRESSURE

vmkernel.log:

vmkernel: cpu) Admission failure in path: host/user/vm.24470028:vmmanon.24470028
vmkernel: cpu) vmmanon.24470028 requires 1812 KB, asked 1812 KB from vm.24470028 which has 166512 KB occupied and 1424 KB available
vmkernel: cpu) Admission failure in path: host/user/vm.24470028:vmmanon.24470028
vmkernel: cpu) vmmanon.24470028 requires 1812 KB, asked 1812 KB from vm.24470028  which has 166512 KB occupied and 1424 KB available

Environment

VMware vCenter Server 8.x
VMware vCenter Server 9.x
VMware ESXi Server 8.x
VMware ESXi Server 9.x

Cause

The vSAN management daemon (vsanperfsvc) process exhausts its allocated memory while processing the QueryPhysicalVsanDisks API call, leading to the observed failures.

Resolution

To resolve this, increase the memory allocated to the vSAN management daemon by modifying an advanced ESXi configuration parameter and then restarting the service.

Steps:

1. SSH to the affected ESXi host.
2. Check allocated memory (default is 174 MB): "esxcfg-advcfg -g /VSAN/VsanMgmtdMaxMemoryInMB"
3. Run the following command to increase the maximum memory limit:
    esxcfg-advcfg -s ### /VSAN/VsanMgmtdMaxMemoryInMB ### changing the limit up to  400MB
4. Restart the vSAN management service:
    /etc/init.d/vsanmgmtd restart
5. Verify the change in the log:
   tail -f /var/log/vsanmgmt.log | grep memory

   vsand[30502911]: [opID=xxxxxxx statsdaemon::_GetNewMemoryMaxLimit] MEM: new max memory limit: 300MB, targetRole=agent, fsEnabled=False, ioDiagEnabled=False

6. Re-run the queryPhysicalVsanDisks API call. It should now complete successfully.

Additional Information

• Reference Document: https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/latest/sdk/vim25/release/HostVsanInternalSystem/moId/QueryPhysicalVsanDisks/post/
• The minimum memory value (eMin = 174 MB) is a hardcoded value and cannot be adjusted.
• Only the maximum memory limit (VsanMgmtdMaxMemoryInMB) can be increased to ensure sufficient memory for vSAN performance and disk query operations.