vSphere 8.0 U3 cluster hosts shows more than two vCLS VMs and consuming disk space

Products

• VMware vSphere ESXi 8.0
• VMware vCenter Server 8.0

Issue/Introduction

• vSphere 8.0 U3 cluster hosts shows more than two vCLS VMs
• The vCLS VMs consumes 2GB+ disk space.
• In ESXi host's /var/run/log/settingsd.log similar errrs as below are seen
  YYYY-MM-DDTHH:MM:SS In(14) settingsd[2099308]: info [ConfigStore:19443d2700] esx/infravisor_pods/vcls hostAPI=VCLS_POD_CRX(ENABLED)
YYYY-MM-DDTHH:MM:SS In(14) settingsd[2099308]: debug [ConfigStore:19443d2700] [GetTaskId]: Generated task uuid <id>
YYYY-MM-DDTHH:MM:SS In(14) settingsd[2099308]: info [ConfigStore:19443d2700] [WriteToFile] Creating directory /var/vmware/lifecycle
YYYY-MM-DDTHH:MM:SSEr(11) settingsd[2099308]: error [ConfigStore:19443d2700] [WriteToFile] Failed to create directory /var/vmware/lifecycle: boost::filesystem::create_directories: File exists [system:17]: "/var/vmware/lifecycle", "/var/vmware"
YYYY-MM-DDTHH:MM:SS Er(11) settingsd[2099308]: error [ConfigStore:19443d2700] [_CheckTaskContention]: Failed to write to the task sync file in task <id>: std::exception, 0 retries left
YYYY-MM-DDTHH:MM:SS Er(11) settingsd[2099308]: error [ConfigStore:19443d2700] Method failed

Environment

• ESXi 8.0.3
• vCenter 8.0.3

Cause

This issue occurs when the scratch location is on /tmp which is non-persistent and is instead located on the /tmp RAMdisk

Resolution

To workaround this issue

• Verify on ESXi host if /var shows a broken symlink.Verify the existence of the following subfolders within /tmp/_osdata####
  • cache
  • core
  • store
  • locker
• If the subfolders are missing,restart the settingsd service using the command /etc/init.d/settingsd restart to recreate the subfolders.

To avoid this issue, Set the Scratch Partition from the vSphere Client