vsanmgmtd process crashes when large number of external LUNs or Disks attached to host.

Products

• VMware vSAN

Issue/Introduction

When a large number of non-vSAN storage devices (such as iSCSI LUNs, storage LUNs, JBOD disks) are attached to a host running vSAN the vsanmgmtd process repeatedly crashes with vsanmgmtd core dumps created.

Log messages seen:
vmkernel.log

2024-06-12T01:01:06.105Z cpu78:32241445 opID=b4eb566a)Admission failure in path: host/vim/vmvisor/vsanperfsvc:vsanmgmtd.32241444:uw.32241444
2024-06-12T01:01:06.105Z cpu78:32241445 opID=b4eb566a)UserWorld 'vsanmgmtd' with cmdline 'vsanmgmtd -s -c /etc/vmware/vsan/vsanmgmt-config.xml'
2024-06-12T01:01:06.105Z cpu78:32241445 opID=b4eb566a)uw.32241444 (251681877) extraMin/extraFromParent: 1792/1792, vsanperfsvc (1765) childEmin/eMinLimit: 41740/43520

vsanmgmt.log

2024-06-12T18:39:59.500Z info vsand[32241444] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=157.444MB, eMinPeak=174.080MB, rMinPeak=174.132MB MEMORY PRESSURE
2024-06-12T19:09:59.678Z info vsand[32241444] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=157.504MB, eMinPeak=174.080MB, rMinPeak=174.132MB MEMORY PRESSURE
2024-06-12T19:39:59.861Z info vsand[32241444] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=157.472MB, eMinPeak=174.080MB, rMinPeak=174.132MB MEMORY PRESSURE

vobd.log

2024-06-08T00:54:50.691Z: [UserWorldCorrelator] 3177574074629us: [esx.problem.application.core.dumped] An application (/usr/lib/vmware/vsan/bin/vsanmgmtd) running on ESXi host has crashed (3 time(s) so far). A core file may have been created at /var/core/vsanmgmtd-zdump.002

Environment

VMware vSAN 6.x
VMware vSAN 7.x
VMware vSAN 8.x

Cause

The vsanmgmtd process performs various checks which perform queries on or about all storage devices attached to the host. When this is a large number (the exact number is not defined) the vsanmgmtd process will run out of memory and crash.

Resolution

Behavior will be improved in a future version to handle this situation better.

There are two workarounds available:

1. Preferred: Reduce the number of storage devices attached to the host.
2. Adjust the allocated memory for the vsanmgmtd process (required on all hosts with non-vSAN storage attached)
   
   1. ssh to the host
   2. Check allocated memory (default is 170 MB): "esxcfg-advcfg -g /VSAN/VsanMgmtdMaxMemoryInMB"
   3. Increase the allocated memory: "esxcfg-advcfg -s 270 /VSAN/VsanMgmtdMaxMemoryInMB"
   4. Check allocated memory (should be 270 MB): "esxcfg-advcfg -g /VSAN/VsanMgmtdMaxMemoryInMB"