Running Scheduled/Manual Sync Depot workflow leads to increased memory consumption of Update Manager service

Products

• VMware vCenter Server

Issue/Introduction

• Multiple services on vCenter might observe out of memory errors and stops responding, for example below sample snippet is for Topology service :
  
  /var/log/vmware/topologysvc/topology-svcs.log

YYYY-MM-DDTHH:MM:SS.###Z [tomcat-exec-86 [] ERROR com.vmware.vapi.protocol.server.msg.json.JsonServerConnection  opId=] Severe internal error
java.lang.OutOfMemoryError: GC overhead limit exceeded
  
  
• vCenter shows high swap memory usage, this can be verified by checking the file /proc/swaps from vCenter SSH session :
  
  root@<hostname> [ ~ ]# cat /proc/swaps
Filename                                Type            Size            Used            Priority
/dev/dm-0                               partition       26206204        15723722          -2
  
  
• Process "updatemgr" shows high memory usage while verifying the stats using "top" command :
  
  PID   USER      PR  NI    VIRT    RES    %CPU  %MEM     TIME+ S COMMAND
<pid> updatem+  20   0 3128508 1714840   0.0   7.9  126:00.06 S /usr/lib/vmware-updatemgr/bin/updatemgr


• As vCenter runs out of overall memory, there are chances of crashes/core files for updatemgr and other services due to high memory consumption.

Environment

• vCenter Server 7.x
• vCenter Server 8.x
• vCenter 9.x

Cause

• Update Manager service when running Sync Depot workflow consumes extra memory than required and does not release it back once the workflow is complete.
• This leads to a gradual increase in Update Manager overall memory usage each time when Sync Depot workflow runs in a scheduled manner.
• Once Update Manager's memory consumption grows large enough, it may cause problems to other running services in vCenter such as memory starvation, service crash/downtime, disk space, etc.

Resolution

This is a known issue with vCenter 8.x / 9.x and Broadcom Engineering is actively working towards fixing this issue in a future patch release version on vCenter 9.x.

Workaround

1. Disable the Automatic Depot Sync by following the procedure described in Configure the vSphere Lifecycle Manager Automatic Download Task to avoid increase in Update Manager memory consumption on a daily basis and do it on demand/manually when ESX LCM workflow needs be performed.
2. Login to vCenter via SSH and restart the Update Manager service to release the extra memory consumed due to previously run Sync Depot operations.

service-control --restart updatemgr

Note: Restarting the service is an optional step and needs to be done if 'updatemgr' service is already using high memory.