Remediation for an Image based cluster failed due to ESX Agent Manager (EAM) - vSphere

Products

• VMware vCenter Server

Issue/Introduction

Remediation for a vSphere Lifecycle Manager (vLCM) Image-based cluster fails because stale ESX Agent Manager (EAM) agency entries are present in the vCenter Server database.

• The vSphere Client UI Cluster updates tab shows the following errors during ESXi host remediation:
  A general system error occurred: Remediation for an Image based cluster failed

'vSphere ESX Agent Manager' service reported issue: Agency ########-############ required solutions [VEE-veecdp] are not properly configured

• Health check reports the following issue: 'vSphere ESX Agent Manager' service reported issue: Agency is unable to determine required solutions from its configured offline depots


• The /var/log/vmware/eam/eam.log file contains entries similar to the following (for example, when using Veeam Continuous Data Protection (CDP) ):

YYYY-MM-DDTHH:MM:SS |  INFO | vim-inv-update | IssueHandlerBase.java | 116 | Updating issues:
New issues:
   []
Removed issues: [

eam.issue.HostNotReachable {
time = YYYY-MM-DDTHH:MM:SS,
description = <unset>,
key = ####,
agency = 'Agency:########-############:null', -> Agency ID
agencyName = 'IoFilter-VEE_bootbank_veecdp',
solutionId = 'VirtualCenter',
solutionName = 'VirtualCenter',
agent = 'Agent:#######################:null',
agentName = '#######################',
host = 'HostSystem:host-#######################',
hostName = 'esxi-example.com',
}

• Stale agencies (such as NSX or vCLS) appear in the ESX Agent Manager pane as "uninstalled but present" or with a "warning" status.

• While looking into vCenter -> Administration -> vCenter Server Extensions -> vSphere ESX Agent Manager Configure Tab shows the following error:

  Offline depot http://url/to/iofilter/zip/package/VMW-some-name-version.zip hosted as https://vcenter-url:443/eam/vib?id=a-guid cannot be uploaded in vSphere Lifecycle Manager

Environment

• vCenter 7.x
• vCenter 8.x

Cause

Stale ESX Agent Manager (EAM) agency entries remain in the vCenter Server database (VCDB) after a manual removal or uninstallation attempt. The EAM service fetches these stale records, causing a mismatch with the current cluster image and blocking remediation.

Resolution

Follow these steps to destroy the stale agency via the Managed Object Browser (MOB).

1. Identify the Agency ID from the error message or the eam.log file (format: Agency:########-############).
2. Navigate to https://<vCenter-FQDN>/eam/mob/?moid=<Agency-ID>.
3. Log in with administrator credentials.
4. Select DestroyAgency.
5. Click Invoke Method.
6. Retry the cluster remediation task in the vSphere Client.

If the MOB method does not resolve the warning:

1. Take a backup or powered-off snapshot of the vCenter Server Appliance.

2. SSH into the vCenter Server as the root user.

3. Stop the vpxd and eam service before making changes to the database:

   • service-control --stop vpxd
   • service-control --stop eam
     
     

4. Connect to the vCenter Server database:

   • /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
     
     

5. Check the entries using the following query:

   • select * from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam';
     
     

6. Note down the stale agency from the above command and remove it from the VCDB:

   • delete from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:#############-#############';
     
     

7. Start the vpxd and eam service.

   • service-control --start vpxd
   • service-control --start eam
     
     

8. Re-try the remediation.

Additional Information

Contact Broadcom Support

Impact/Risks:

• Remediation fails for Image Based clusters.
• vCLS agents will not get deployed.
• vCLS health will be in degraded state.