vsanmgmtd MemoryError during High Resolution Performance Stats Rollup - VMware vSAN

Products

• VMware vSAN

Issue/Introduction

The vsanmgmtd service may crash on ESXi hosts in a vSAN cluster when the vSAN Performance Service is running in High Resolution Mode against a large stats object. This issue manifests as a MemoryError and an admission failure in the VMkernel logs.

Symptoms:

• The vsanmgmtd service crashes and restarts on ESXi hosts.
• The following error appears in /var/run/log/vsanmgmtd.log:
  

  Er(11)[+] vsand[#####]: MemoryError
  Er(11) vsand[#####]: [opID=MainThread vsanperfsvc::main] vsan mgmt daemon start fail.

• The following admission failure appears in /var/run/log/vmkernel.log, where the system indicates that the vsanperfsvc resource group has insufficient memory to accommodate the daemon's allocation request:

  2026-10-06T23:51:03.194Z In(182) vmkernel: cpu36:####) uw.#### requires 256 KB, asked 256 KB from vsanperfsvc (2073) which has 208700 KB occupied and 196 KB available.
  2026-10-06T23:51:03.194Z In(182) vmkernel: cpu36:####) Admission failure in path: host/opt/vsan/vsanperfsvc:vsanmgmtd.####:uw.####

Environment

• VMware vSAN 8.x

 

Cause

When the vSAN Performance Service is configured in High Resolution Mode against a large stats object (e.g., 512 GB), the baseline memory footprint of the vsanmgmtd daemon increases to approximately 145–150 MB. The default memory limit for the parent UserWorld scheduler group (vsanperfsvc) is capped at 204 MB. During scheduled performance data rollups, temporary memory allocations can cause the total group consumption to exceed this hard ceiling, resulting in a Python MemoryError and process termination.

The following logs can be checked to confirm that the vSAN performance service is configured in High Resolution Mode against a large stats object.

vsand[####]: [opID=MainThread statsdaemon::_preCheckForEnablingHighResolutionMode] _preCheckForEnablingHighResolutionMode, preCheckPassed => True, isVsanEnabled => True, self.perfEnabled => True, isStatsObjectExisting => True, currentObjSize => 549755813888

Reference: High Resolution Performance Monitoring in vSAN 8 U1

Resolution

To resolve this issue, increase the memory limit for the vsanmgmtd service to 270 MB across all ESXi hosts in the cluster.

1. Log in to the ESXi host via SSH.
   
   
2. Check the current allocated memory limit (the default is typically 204 MB):
   esxcfg-advcfg -g /VSAN/VsanMgmtdMaxMemoryInMB


3. Set the memory allocation limit to 270 MB:
   esxcfg-advcfg -s 270 /VSAN/VsanMgmtdMaxMemoryInMB


4. Confirm the new setting is active:
   esxcfg-advcfg -g /VSAN/VsanMgmtdMaxMemoryInMB


5. Restart the management service so it applies the higher memory ceiling:
   /etc/init.d/vsanmgmtd restart

Note: These settings survive host reboots. Ensure these commands are executed on every host in the vSAN cluster to maintain consistency.

Additional Information

• vsanmgmtd process crashes when large number of external LUNs or Disks attached to host.
• vSAN API QueryPhysicalVsanDisks() may fail with MemoryError.
• High Resolution Performance Monitoring in vSAN 8 U1
• For instructions on gathering diagnostic data, see Collecting diagnostic information for VMware ESXi.
• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.
• For future updates regarding fix status or enhanced automation for this workflow, we recommend subscribing to this article: Knowledge Article Subscription Guide