Configuring vSphere HA on an image-based cluster fails.

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Enabling vSphere HA on the cluster fails with the below errors

"A general system error occurred: Installing HA Components failed on the cluster: domain-cxx".

Cannot complete the configuration of the vSphere agent on the host. "Applying HA VIBs on the cluster encountered a failure". Failed installing HA component on the host: host-xxxx.

• Error in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[#######] [Originator@#### sub=VumVapi::Utils opID=01b95dec-5b0c-####-####-0b0e6d9bae95] [DepotContentManager 696] Failed to get cached component. No record - VCIDB ERROR: Row with primary key (vsphere-fdm, #.#.#-########) not found in table PM_DEPOT_COMPONENTS

Note :  #.#.#-######## notation is for the vsphere-fdm version which is different for each vCenter build number.

• Error in /var/log/vmware/vmware-updatemgr/vum-server/imageservice.log

INFO imageService[1401######22048] [SoftwareSpecMgr 1368] Image validation result: {'info': [], 'warnings': [], 'errors': [{'id': 'com.vmware.vcIntegrity.lifecycle.EsxImage.ComponentNotFoundError', 'message': {'id': 'com.vmware.vcIntegrity.lifecycle.EsxImage.ComponentNotFoundError', 'default_message': 'Component vsphere-fdm cannot be found in depot.', 'args': ['vsphere-fdm']}, 'resolution': None, 'time': 'YYYY-MM-DD'}]}  vmware.esximage.Errors.ComponentNotFoundError: ('vsphere-fdm', '#.#.#-########', 'Could not find the component with name = vsphere-fdm, version = #.#.#-######## in the depot.')

• The ESXi host where the vSphere HA configuration is failing will have a older version of the vsphere-fdm agent present, which can be confirmed using the following command after SSHing into the ESXi host as the root user ( The build number of the vCenter and the vsphere-fdm vib should match ) : 

esxcli software vib list | grep fdm

• Validating an image at cluster level on the vCenter results in "Image Validation Failed"

Environment

• VMware vCenter Server 8.x
• VMware vSphere ESXi 8.x
• VMware vCenter Server 7.x
• VMware vSphere ESXi 7.x

Cause

The vCenter is not able to get the cached component ( vsphere-fdm vib ) from the VCDB of the updatemgr service psql table PM_DEPOT_COMPONENTS, after vCenter update/upgrade.

Resolution

1. Take snapshot of vCenter VM (Powered off snapshot of all vCenter Servers if in ELM)

2. SSH to vCenter Server with root credentials

Enter the shell with below command

shell

3. Stop the update manager service

service-control --stop vmware-updatemgr

4. Access the postgres DB

su updatemgr -s /bin/bash

psql -U vumuser -d VCDB

5. View the below two tables before deleting the required entries

table pm_software_desired_states;

table pm_software_compliances;

6. Delete the entries from the same two tables if all clusters experience the same issue or there is only 1 cluster in the environment

DELETE FROM pm_software_compliances;

DELETE FROM pm_software_desired_states;

6.a If there is more than 1 cluster in your environment and only 1 cluster is impacted

In the UI select the cluster from inventory

Note the cluster domain ID from the URL of the browser. It should be similar to domain-cXXXX: exclude everything after the : 

DELETE FROM pm_software_compliances where desired_state_id in (select desired_state_id from pm_software_desired_states where entity_id='domain-cXXX');

 

DELETE FROM pm_software_desired_states where entity_id='domain-cXXX';

7. Quit and exit from the DB:

\q

8. Start shell session as root user

su root -s /bin/bash/

9. Start the update manager service

service-control --start vmware-updatemgr

10. Rebuild the cluster image from UI and enable the vSphere HA.