VCSA services crash with Out of Memory due to updatemgr

Products

• VMware vCenter Server

Issue/Introduction

Services running on the VCSA report that they have ran out of memory.

• vpxd will crash with "Unable to allocate memory"
• vpxd-svcs, vmware-sps, vapi endpoint services will observe a java_error<pid>.log file reporting for the service
  
  Out of Memory Error (os_linux.cpp:2797), pid=#####, tid=0x0000###########
  
  
• sso services observe a hs_err_sts_pid######.log file
  
  #  Out of Memory Error (os_linux.cpp:2797), pid=#####, tid=0x0000###########
  
  
• meminfo will report High Committed_AS utilization
  
  • grep Committed_AS /proc/meminfo
    
    
• cloudvm-ram-size-output and procstate will report high utilization of the updatemgr service
  

  • ========= Start of cloudvm ram size dump at Tue Feb 18 08:01:01 AM UTC 2025 ==========

    /usr/sbin/cloudvm-ram-size -S
Service-Name            AllocatedMB     MaxMB CurrentMB  Curr-RSS    Cache  MapFiles   MemoryLimit
...
updatemgr                        -1     23507     13367     11523       33        19 8796093022207 <-------
vapi-endpoint                    -1       752       591       558       21         0 8796093022207
vc-ws1a-broker                 2048        17        10         8        1         1 8796093022207
vgauthd                          -1        10         8         1        8         7 8796093022207
vlcm                            202        76        31        21        8         8 8796093022207
vmtoolsd                         -1         7         4         3        1         1 8796093022207
vmware-pod                      200       200        58        42       12         9           200
vmware-postgres-archiver          5      4180      2540         0     2454         0 8796093022207
vmware-vdtc                      35        35        34         2       30         0            35
vmware-vmon                       5       497        42        10       26         9 8796093022207
vmware-vpostgres               4275     28629     21853       813    20025      2217 8796093022207
...
TOTAL(RAM=58241.57421875MB)     14980    117725    108120     27078    25089      2550 8796093022207
========== End of cloudvm ram size dump ==========

  • ========== Start of system state dump at Tue Feb 18 08:01:01 AM UTC 2025 ==========

    USER        PPID     PID %CPU %MEM    VSZ   RSS TTY      STAT START     TIME  MAJFL  MINFL COMMAND
updatem+    ####     ##### 3.5 19.0 25917120 11342328 ?   Sl    2024 2-08:38:42 14154808 843000337 /usr/lib/vmware-updatemgr/bin/updatemgr /usr/lib/vmware-updatemgr/bin/vci-integrity.xml <-------

    ========== End of system state dump ==========

Environment

vCenter Server 7.x
vCenter Server 8.x

Cause

This is due to a memory leak in updatemgr.

Resolution

This is a known issue currently affecting VMware Update Manger and will be resolved in a future product release.

For vCenter Server 8.x, this issue has been resolved in 8.0U3e, see VMware vCenter Server 8.0 Update 3e Release Notes

Workaround

Restart the updatemgr service periodically to release memory.

1. SSH to the vCenter via root
   
   
2. Restart the updatemgr service
   
   vmon-cli -r updatemgr