vCLS Machines in Orphaned State Due to EAM Login Failure

Products

• VMware vCenter Server

Issue/Introduction

The vSphere Cluster displayed the following error on the Summary page:

“vSphere DRS functionality was impacted due to an unhealthy state of vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS.”

Multiple vCLS virtual machines were found in an orphaned state within the affected cluster.
Upon reviewing the eam.log, the following recurring error was observed:

“Internal server error during dispatch” and “EAM is still loading from database. Please try again later.”

Environment

vCenter 7.x

vCenter 8.x

Cause

ESX Agent Manager (EAM) failed to authenticate with vCenter due to an expired or invalid vpxd-extension certificate. This prevented EAM from managing vCLS VMs properly, resulting in the cluster service health degradation.

Resolution

To resolve the issue, the vpxd-extension certificate of the vCenter Server Appliance was updated using the steps below:

1. Log in to the vCenter Server Appliance using SSH. 
2. Run this command to retrieve the vpxd-extension solution user certificate and key:
   
   mkdir /certificate
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
    
3. Edit the "<vCenter_Server_Hostname>" in the below command and run this command to update the extension's certificate with vCenter Server.
   
   python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <vCenter_Server_Hostname> -u [email protected]
   
   Note: The default user and domain is [email protected]. Change the domain to match the environment's vCenter SSO. When prompted, type in the [email protected] password.
   
   
4. Restart the VMware ESX Agent Manager service with these commands: 
   
   service-control --stop vmware-eam
   service-control --start vmware-eam

NOTE: If the above steps do not resolve the issue, we will need to regenerate new solution user certificates by using option 6 from the certificate manager tool. Ref: Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA