vSAN Management (vsanmgmt service) memory leak in ESXi 7.0 U1 and 7.0U2

Products

• VMware vSphere ESXi

Issue/Introduction

Identify and resolve issues with ESXi host(s) hitting a vsanmgmt memory leak.

Symptoms:
You experience these symptoms on an ESXi 7.0 U1 or 7.0U2 vSAN enabled host

• Physical disk health retrieval issues with error Host XXX - Error: Out of Memory
• Advanced vSAN Configuration in sync - vSAN daemon liveness - CLOMD status: Abnormal
• Inaccurate health check results due to vsanmgmtd memory leak

In the /var/run/log/vmkernel.log files on the ESXi host, you see messages similar to:  

• 2021-01-20T09:33:12.993Z cpu40:13487007)Admission failure in path: host/vim/vmvisor/vsanperfsvc:cmmds-tool.13487007:uw.13487007
• 2021-01-20T09:33:12.993Z cpu40:13487007)Admission failure in path: host/vim/vmvisor/vsanperfsvc:cmmds-tool.13487007:uw.13487007

In the var/run/log/vsanmgmt.log file, you see messages similar to:  

• 2021-01-20T03:33:55.754Z info vsand[2510678] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=171.324MB, eMinPeak=173.968MB, rMinPeak=174.984MB  MEMORY PRESSURE
• 2021-01-20T04:03:59.304Z info vsand[2510678] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=170.428MB, eMinPeak=173.968MB, rMinPeak=174.984MB  MEMORY PRESSURE

In the /var/run/log/vmsyslogd-dropped.log file, you see messages similar to:  

• 2020-09-25T04:38:38.257Z: 2020-09-25T04:38:38.021Z cpu51:18265550 opID=34b30f7d)Admission failure in path: host/vim/vmvisor/vsanperfsvc:vsanmgmtd.18265531:uw.18265531

Note: The timestamp and memory usage will vary for different clusters.

The vsanmgmtd core dump file (vsanmgmtd-zdump.XXX) can be found under /var/core/

Environment

VMware vSphere ESXi 7.0.x

Cause

Memory leak issue in vsanmgmtd service affecting ESXi 7.0 U1 and 7.0U2. This issue only affects the vSAN management stack.

Resolution

This issue is resolved by upgrading both vCenter and ESXi to 7.0U3c or later.

Workaround:
If you are unable to upgrade, you can work around the issue by restarting the vsanmgmt service on the affected ESXi host(s) via SSH: 

# /etc/init.d/vsanmgmtd restart

Additional Information

Impact/Risks:
If the vsanmgmtd encounters a memory leak it may cause inaccurate health check results which are only resolved by restarting vsanmgmt or rebooting the ESXi Host.

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: ESXi
• Log File: vmkernel.log
• Log Expression Check "Admission failure in path: host/vim/vmvisor/vsanperfsvc:vsanmgmtd"