vCLS VMs are not getting deployed or when deployed they are in powered off state

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• vCLS VMs are not getting deployed on ESXi hosts
• On few clusters if they are deployed they are in powered off state.
• If they are attempted to power on, there is an error "Another task is already in progress"
• Some clusters shows an error " vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS"
• In ESXi host's /var/run/log/settingsd.log similar error as below is seen
  YYYY-MM-DDTHH:MM:SS In(14) settingsd[2099308]: info [ConfigStore:19443d2700] esx/infravisor_pods/vcls hostAPI=VCLS_POD_CRX(ENABLED)
YYYY-MM-DDTHH:MM:SS In(14) settingsd[2099308]: debug [ConfigStore:19443d2700] [GetTaskId]: Generated task uuid <id>
YYYY-MM-DDTHH:MM:SS In(14) settingsd[2099308]: info [ConfigStore:19443d2700] [WriteToFile] Creating directory /var/vmware/lifecycle
YYYY-MM-DDTHH:MM:SSEr(11) settingsd[2099308]: error [ConfigStore:19443d2700] [WriteToFile] Failed to create directory /var/vmware/lifecycle: boost::filesystem::create_directories: File exists [system:17]: "/var/vmware/lifecycle", "/var/vmware"
YYYY-MM-DDTHH:MM:SS Er(11) settingsd[2099308]: error [ConfigStore:19443d2700] [_CheckTaskContention]: Failed to write to the task sync file in task <id>: std::exception, 0 retries left
YYYY-MM-DDTHH:MM:SS Er(11) settingsd[2099308]: error [ConfigStore:19443d2700] Method failed

Environment

• ESXi 8.x
• vCenter 8.x

Cause

This issue occurs when the scratch location is on /tmp which is non-persistent and is instead located on the /tmp RAMdisk

Resolution

To workaround this issue

• Verify on ESXi host if /var shows a broken symlink.
• Verify the existence of the following subfolders within /tmp/_osdata####
  • cache
  • core
  • store
  • locker
• If the subfolders are missing, restart the settingsd service using the  command /etc/init.d/settingsd restart to recreate the subfolders.

To avoid this issue, Set the Scratch Partition from the vSphere Client