vCenter High Memory Utilization Due to /tmp Directory being full

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server Appliance Management Interface has alert "Appliance is running low on memory".
  
  
  
  
• Hosts are configured to boot using Auto Deploy.
• /tmp directory on vCenter Server reaches 100% disk usage, which triggers high memory alert.
• After clearing /tmp directory in vCenter Server, memory utilization drops back to normal levels.
• In /var/log/vmware/rbd/rbd-syslog.log, the following entries are seen.
  

  YYYY-MM-DD:T:HH:MM:SS [1368581:CacheWorker-1]ERROR:vibutil:Errors encountered while repairing cache: Failed to repair the cache: Something went wrong while converting items to pxe profile:No profiles found with the given name
YYYY-MM-DD:T:HH:MM:SS [1368581:CacheWorker-1]ERROR:pluginmaster:exception:rbdplugins.plainpxe.vmwCacheRequest -- [Errno 28] No space left on device
Traceback (most recent call last):
 File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry

Environment

• VMware vCenter Server 8.x
• VMware vSphere ESXi 8.x

Cause

• While populating the Auto Deploy VIB cache, vCenter Server might initiate multiple requests to get the same VIB file which might lead to an overload of the /tmp folder space. As a result, vCenter Server becomes unresponsive and regular tasks such as deploying a virtual machine or creating a backup might fail.
• vCenter Server tmp directory are filled with below files
  
  ls -lrth /tmp | grep -E 'tmp|deploy'

-rw-------   1 deploy  deploy      0 MM DD MM:SS tmp2dbt9uon
drwx------   5 deploy  deploy    100 MM DD MM:SS tmppq3c5_0p
drwx------   2 deploy  deploy     80 MM DD MM:SS tmpdhr2w8o6
-rw-------   1 deploy  deploy      0 MM DD MM:SS tmpeid4umdb

Resolution

This issue is resolved in VMware vCenter Server 8.0 Update 3g Build 24853646. Refer to VMware vCenter 8.0 Update 3g Release Notes

As a workaround , implement the steps below

Before proceeding with the steps below, take a snapshot of vCenter.

• Delete rules and image profiles from Autodeploy  that are not in use.
• Delete files from the tmp folder that are owned by the deploy user and have filenames starting with "tmp". 

Additional Information

• For more information on full vCenter Server tmp directory, refer to the article 'vCenter Server Appliance disk space is full'