Remediation for an Image based cluster fails: 'vSphere ESX Agent Manager' reported issue: Agency xxxxxxx solutions from its configured offline depots

Products

• VMware vCenter Server

Issue/Introduction

• Image based remediation failing due to stale EAM agent manager entries in the VCDB
• Messages similar to below can be found in /var/log/vmware/eam/eam.log

YYYY-MM-DDTHH:MM:SS |  INFO | vim-inv-update | IssueHandlerBase.java | 116 | Updating issues:
New issues:
   []
Removed issues: [

eam.issue.HostNotReachable {
time = YYYY-MM-DDTHH:MM:SS,
description = <unset>,
key = xx,
agency = 'Agency:XXXXXXXX-XXXXXXXXXXX:null',
agencyName = 'IoFilter-VEE_bootbank_veecdp',
solutionId = 'VirtualCenter',
solutionName = 'VirtualCenter',
agent = 'Agent:xxxxxxxxxxxxxxxx:null',
agentName = 'xxxxxxxxxxxxxxxx',
host = 'HostSystem:host-xxxxxxxxxxxxx',
hostName = 'esxi-name.com',
}

• While looking into vcenter -> Administration -> vCenter Server Extensions -> vSphere ESX Agent Manager Configure Tab shows the following error:
  Offline depot http://url/to/iofilter/zip/package/VMW-some-name-version.zip hosted as https://vcenter-url:443/eam/vib?id=a-guid cannot be uploaded in vSphere Lifecycle Manager.
• Running precheck or cluster upgrade in a vSphere 8.X/ 7.X, while using vSphere Cluster image runs into the following issue:
  Health Check for 'Cluster name' failed
'vSphere ESX Agent Manager' service, running on 'cluster 'Cluster name'', reported issue: Agency <UUID of  Agency> is unable to determine required solutions from its configured offline depots
'vSphere ESX Agent Manager' service, running on 'cluster 'Cluster name'', reported issue: Agency <UUID of  Agency> is unable to determine required solutions from its configured offline depots
• Reset on VUM DB has been tried, cluster image correct/  imported from working cluster etc 

Environment

8.0.x

Cause

Stale entries of the old agents was being fetched by the EAM service from the VCDB.

Resolution

Option 1:

1. Go to https://<vcsa-dns-name>/eam/mob/?moid=<agency_id>
<vcsa-dns-name> is the VC FQDN.
<agency_id> is the ID of the agency that needs to be destroyed (This is retrieved from the error message )

2. Click DestroyAgency 

3. A new window will be opened and there you have to click Invoke Method.

4. Retry the patch operation 

Option 2: This involves editing the database.

NOTE: Before applying the steps below, please take a backup or snapshot of the vCenter Server Appliance . If the vCenter is part of a Linked Mode replication setup, please backup/powered off snapshots of all replicating nodes as well.
 

• Stop the vpxd and eam service before making changes to the database
  • service-control --stop vpxd
  • service-control --stop eam
• Connect to the database
  • /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
• Check the entries 
  • select * from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam';
• Note down the stale agency from the above command and remove it from the VCDB
  • delete from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:XXXXXXXX-XXXXXXXXXXX';
• Start the vpxd and eam service
  • service-control --start vpxd
  • service-control --start eam
• Re-try the remediation again

Additional Information

Impact/Risks:

• Remediation fails for Imaged Based clusters
• vCLS agents will not get deployed.
• vCLS health will be in degraded state.