WCP Supervisor Cluster struck in "Removing" state with error "Failed to get EAM agencies"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• WCP Supervisor Cluster will be struck in removing state for a long time
• In the /var/log/vmware/wcp/wcpsvc.log, you may find entries similar to:

Supervisor control plane failed: No connectivity to API Master: connectivity , config status REMOVING
Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later.
Failed to list EAMAgent. Err ServerFaultCode: EAM is still loading from database. Please try again later.

• In the /var/log/vmware/eam/eam.log, you may find entries similar to:

|  WARN | vim-async-0 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:a6307ae03f141f9d] Re-login failed, due to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.

Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

| ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.

Environment

VMware vCenter Server

Cause

WCP is dependent on EAM service for various operations including deletion of supervisor cluster. This issue occurs when the EAM service is not able to authenticate to vCenter due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension.

Resolution

Update the certificate of the EAM extension in VPXD using the below procedure:

• Download the vCert script by referring to below article and upload it the /tmp directory on vCenter

vCert - expired certificate replacement script

• Execute the script using below commands

# unzip -q vCert-6.0.1-20250516.zip
# cd vCert-6.0.1-20250516
# chmod +x vCert.py
# ./vCert.py

• Select option 3: Manage certificates
• Select option 6:vCenter Extension thumbprints
• The script will check and report thumbprint mismatches

Example:

For this specific issue, EAM service will report as MISMATCH

Note: There may be other extensions such as image builder and rbd reporting mismatch

• At the prompt enter "y" and hit enter to update the extension thumbprints

Note: All the extensions which are reported as MISMATCH are updated and the respective services will be restarted

Example output:

• Use option 6:vCenter Extension thumbprints to verify if the extensions are updated
• WCP cluster removal will complete and we will no longer see authentication errors in EAM logs