vCLS VMs are not getting created in vSphere 8.0u3

Products

• VMware vCenter Server

Issue/Introduction

vCLS virtual machines do not deploy in vSphere 8.0u3

No tasks related to vCLS deployment are observed in the vSphere Web Client.
Triggering Enter or Exit Retreat Mode does not initiate the deployment of new vCLS VMs.
Additionally, the ESX Agent Manager log on the vCenter (/var/log/vmware/eam/eam.log) does not report any errors
In /var/log/vmware/vpxd.log we see the following 

####-##-##T##:##:## warning vpxd[19460] [Originator@6876 sub=MoCluster opID=PodCrxMgr-domain-c#######,] Failed to disable vCLS Pod Crx; [vim.HostSystem:host-#####,hostfqdn], [vim.ClusterComputeResource:domain-c#######], N4Vpxd7Clients28InternalServerErrorExceptionE(Error:

-->    com.vmware.vapi.std.errors.internal_server_error

--> Messages:

-->    com.vmware.esx.task.create.error<Failed to create task. Please retry later>

In ESXi Host -> Monitor -> Events tab, the alarm "Duplicate name 'vmw.#####.json' entry in cache." triggers frequently.

In  /var/lib/vmware/osdata/healthd, you can find the below files created (the number of files may differ):

vmw.#####-1.json
vmw.#####-2.json
vmw.#####-3.json

Environment

vCenter Server 8.0u3
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

 

Cause

Multiple LUNs are being referenced by the OSDATA partition on the ESXi host

Resolution

1. Take an SSH session to the ESXi Host

2.Execute the following command to display information about VMFS datastores and extent
[root@esxi :~ ] esxcli storage vmfs extent list
Volume Name                                        VMFS UUID                              Extent Number     Device Name                  Partition
OSDATA-6672####-b565####-a412-005056a####         6672####-b565####-a412-005056a####     0                px. vmhba0: CO : TO : LO     7
OSDATA-1127####-b768####-b987-002326a####         6645####-b532####-a412-002526a####                       px. vmhba0: CO : TO : L1     7

[root@esxi:~] df -h
Filesystem  Size   Used Available Use% Mounted on
VMFSOS     23.8G   4.3G     19.4G  18% /vmfs/volumes/OSDATA-6672####-b565####-a412-005056a####
VMFS01      12.7G   2.3G      10.4G     18%    /vmfs/volumes/OSDATA-1127####-b768####-b987-002326a####  
vfat        4.0G 258.0M      3.7G  6% /vmfs/volumes/BOOTBANK1
vfat        4.0G 335.4M      3.7G  8% /vmfs/volumes/BOOTBANK2
vsan        2.7T   2.4T    359.5G  87% /vmfs/volumes/VCF-GS-Cloud-vSAN01

Once the incorrect volume mapping to the OSDATA partition was removed, the ESXi host functioned as expected and the vCLS VMs were successfully deployed without further issues.