vCenter root disk exhaustion warning alarm on vCenter post 8.0 u3 patch.

Products

• VMware vCenter Server

Issue/Introduction

• After patching to vCenter Server to 8.0 U3, a warning alarm for root disk exhaustion is triggered.
• Reviewing the df -h shows that root partition /overlay is full/almost 100% on vCSA 

Filesystem	Size	Used	Avail	Use%	Mounted on
devtmpfs	4.0M	0	4.0M	0%	/dev
tmpfs	29G	1012K	29G	1%	/dev/shm
tmpfs	12G	1.4M	12G	1%	/run
tmpfs	4.0M	0	4.0M	0%	/sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0	47G	47G	0G	100%	/
/dev/sda3	488M	37M	415M	9%	/boot
/dev/sda2	10M	2.0M	8.1M	20%	/boot/efi
tmpfs	29G	4.6M	29G	1%	/tmp
/dev/mapper/autodeploy_vg-autodeploy	25G	8.6M	24G	1%	/storage/autodeploy
/dev/mapper/lifecycle_vg-lifecycle	98G	4.3G	89G	5%	/storage/lifecycle
overlay	47G	47G	0G	100%	/storage/containers/vc-ws1a-broker/27###########################e78/rootfs

Environment

vSphere vCenter Server 8.0 U3 and above

Cause

• At the time of loading the containers, vCenter will take snapshot of the containers. This approximately takes 2-3 GB of space in root partition. When the patching of vCenter occurs, the snapshot path grows to double in size leaving stale files and thereby keep on increasing the space it occupies in the vCenter. 

 

Resolution

• Workaround:

NOTE: Please backup vCenter as per best practices before proceeding further.

1. Download attached remove_stale_overlayfs_snapshot.py

2. Use the file-moving utility of your choice (WinSCP for example) to copy the remove_stale_overlayfs_snapshot.py to the affected vCenter.

NOTE:  If trouble is encountered while connecting to a vCenter appliance using WinSCP, please see Error when uploading files to vCenter Server Appliance using WinSCP

3. Assign permissions to the file 

chmod +x remove_stale_overlayfs_snapshot.py 

4. Run the command python remove_stale_overlayfs_snapshot.py 

• Permanent Fix:

Upgrade to VMware vCenter Server 8.0 Update 3g

Note: If updating the vCenter is failing with errors in rbd logs as below please refer to Auto deploy is not booting hosts and web ui is not responding

 

Something went wrong while converting items to pxe profile:No profiles found with the given name
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vmodl_impl/pxe_profile_util.py", line 151, in ConvertToPxeProfile
rbd.vmodl_impl.pxe_profile_util.ProfileMiss: No profiles found with the given name
[611717:CacheWorker-0]ERROR:pxe_profile_util:Failed to repair the cache: Something went wrong while converting items to pxe profile:No profiles found with the given name
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vmodl_impl/pxe_profile_util.py", line 151, in ConvertToPxeProfile
rbd.vmodl_impl.pxe_profile_util.ProfileMiss: No profiles found with the given name