Applying personality to cluster domain-cx failed. Lifecycle manager reports - vSphere ESX Agent Manager service, running on 'host' [Host], reported issue.

Products

• VMware SDDC Manager

Issue/Introduction

• Adding a new host to a cluster in the SDDC manager fails at subtask - Remediate ESXi Host(s) to be Compliant with Cluster's Image.

Description:

Remediate ESXi Host(s) to be Compliant with Cluster's Image

Progress Messages:

Applying personality to cluster domain-cx failed with error: com.vmware.vapi.std.errors.Error
Error (com.vmware.vapi.std.errors.error){
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message)
{id = com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed,
defaultMessage = Health Check for '[Host]' failed, args = [Host],
params = <null>,
localized = Health Check for '[Host]' failed }],
data = <null>,
errorType = ERROR }

 

• Checking Updates tab for the cluster on the vCenter server, Remediation result shows the task failed on the host we are trying to add to the cluster.
  
  'vSphere ESX Agent Manager' service, running on 'host' [Host], reported issue: System VM on host HostSystem:host-xxxx required by Agent 91df5e3c-xxxx-xxxx-xxxx-xxx is not ready due to a host in Maintenance Mode
  
  

 

Environment

VMware Cloud Foundation 5.x

Cause

Issue with ESX agents fetched by the EAM service on vCenter server.

Resolution

There are 2 ways to resolve this issue.

1. Restart the management agents on the ESXi host and manually try to remediate the cluster from the vCenter. Once the hosts are complaint with the image, re-start the task from the SDDC manager.
   
   OR 
   
   
2. Delete the Agency from the vCenter Database. 

^{NOTE: Before applying the steps below, please take a backup or snapshot of the vCenter Server Appliance . If the vCenter is part of a Linked Mode replication setup, please backup/powered off snapshots of all replicating nodes as well.}

1. Stop the vpxd and eam service before making changes to the database

service-control --stop vpxd

service-control --stop eam

2. Connect to the database

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

3. Check the entries 

select * from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam';

4. Note down the problematic agency from the above command referring to the agency ID from the remediation error (host-xxxx required by Agent 91df5e3c-xxxx-xxxx-xxxx-xxx is not ready due to a host in Maintenance Mode). Remove it from the VCDB

delete from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:XXXXXXXX-XXXXXXXXXXX';

5. Start the vpxd and eam service

service-control --start vpxd

service-control --start eam

6. Re-start the task to add host from the SDDC manager.