vCLS Not Getting Deployed due to "Found inconsistency in the DB! There is no VM factory referred by agency HostVMAgency"

Products

• VMware vCenter Server

Issue/Introduction

• The following error is displayed on the cluster object after vCenter is upgraded:
  
  vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS.
  
  
• vCLS VMs are failing to deploy.
  
  
• In vCenter /var/log/vmware/eam/eam.log you see the following entries:
  

  YYYY-MM-DDTHH:MM:SS | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.

  YYYY-MM-DDTHH:MM:SS | INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.db.PersistenceException: Found inconsistency in the DB! There is no VM factory referred by agency HostVMAgency(ID:'Agency:##########:null')

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

Stale ESX Agent Manager (EAM) agent entries are present in the vCenter Server database (VCDB). When the EAM service attempts to fetch these old entries during initialization, it encounters a database inconsistency that prevents the deployment of new vCLS agents

Resolution

 Follow these steps to remove stale EAM agent entries from the vCenter Server database.

Note: Take a snapshot of the vCenter (If this is an enhanced linked mode environment, take powered-off snapshots for consistency).

1. Connect to the vCenter Server via SSH as the root user.
   
   
2. Stop the services that interact with the ESX Agent Manager to prevent database conflicts:
   
   service-control --stop content-library
service-control --stop eam
service-control --stop vpxd
   
   
3. Connect to the vCenter database (VCDB) using the following command:
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

4. Check the VCDB for the old agent entries:
   
   select * from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:##########';
   
   
5. Once the agent entry on the database has been verified, remove entries with the following command :
   
   delete from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:##########';
   
   
6. Exit the database: 
   \q
7. Start the services:
   
   service-control --start vpxd
service-control --start eam
service-control --start content-library