Remediation task for an Image based cluster fails with Error: "Remediation for an Image based cluster failed"

Products

• VMware vCenter Server

Issue/Introduction

• vSphere client UI's Cluster updates tab shows the following error message when the ESXI host remediation task fails:

A general system error occurred:

Remediation for an Image based cluster failed

'vSphere ESX Agent Manager' service, running on 'cluster '<Cluster name>'', reported issue: Agency <Agency_id> is unable to determine required solutions from its configured offline depots

Health Check for '<Cluster name>' failed

• Error entries similar to below can be found in /var/log/vmware/eam/eam.log file:

YYYY-MM-DDTHH:MM:SS |  INFO | vim-inv-update | IssueHandlerBase.java | 116 | Updating issues:
New issues:
   []
Removed issues: [

eam.issue.HostNotReachable {
time = YYYY-MM-DDTHH:MM:SS,
description = <unset>,
key = xx,
agency = 'Agency:########-############:null', -> Agency_id
agencyName = 'IoFilter-VEE_bootbank_veecdp',
solutionId = 'VirtualCenter',
solutionName = 'VirtualCenter',
agent = 'Agent:#######################:null',
agentName = '#######################',
host = 'HostSystem:host-#######################',
hostName = 'esxi-example.com',
}

• While looking into vCenter -> Administration -> vCenter Server Extensions -> vSphere ESX Agent Manager Configure Tab shows the following error:

  Offline depot http://url/to/iofilter/zip/package/VMW-some-name-version.zip hosted as https://vcenter-url:443/eam/vib?id=a-guid cannot be uploaded in vSphere Lifecycle Manager.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

Stale entries of the old agents was being fetched by the EAM service from the VCDB.

Resolution

Option 1:

1. Go to https://<vcsa-dns-name>/eam/mob/?moid=<Agency_id>

<vcsa-dns-name> is the VC FQDN.
<Agency_id> is the ID of the agency that needs to be destroyed (This can be retrieved from the error message )

2. Click DestroyAgency 

3. A new window will be opened and there you have to click Invoke Method.

4. Retry the patch operation 

Option 2: 

Important: This option involves editing the vCenter Server database, before applying the steps outlined below, take a backup or snapshot of the vCenter Server Appliance; If the vCenter is part of a Linked Mode replication setup, take backup/powered off snapshots of all replicating nodes.

1. SSH into the vCenter Server as the root user.

2. Stop the vpxd and eam service before making changes to the database:

service-control --stop vpxd

service-control --stop eam

3. Connect to the vCenter Server database:

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

4. Check the entries using the following query:

select * from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam';

5. Note down the stale agency from the above command and remove it from the VCDB:

delete from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:#############-#############';

6. Start the vpxd and eam service.

service-control --start vpxd

service-control --start eam

 

7. Re-try the remediation again.

Additional Information

Impact/Risks:

• Remediation fails for Imaged Based clusters.
• vCLS agents will not get deployed.
• vCLS health will be in degraded state.