vCLS Not Getting Deployed due to "Found inconsistency in the DB! There is no VM factory referred by agency HostVMAgency"

Products

• VMware vCenter Server

Issue/Introduction

• The following error is displayed on the cluster object after vCenter is upgraded to 7.x
  
  vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS.
  
  
• vCLS VMs are failing to deploy.

Environment

VMware vCenter Server 7.0.0

Cause

There are stale entries of the old EAM agents being fetched by the EAM service from the vCenter database. The service is then failing to deploy new vCLS agents. This can be validated with the following messages in vCenter - /var/log/vmware/eam/eam.log:

2022-09-01T22:32:36.591Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch

com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.

2022-09-01T22:32:32.586Z | INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.db.PersistenceException: Found inconsistency in the DB! There is no VM factory referred by agency HostVMAgency(ID:'Agency:##########:null')

 

Resolution

Deleted the old EAM agents from the vCenter database:

Pre-requisite: Please snapshot the vCenter (If this is an enhanced linked mode environment, powered off snapshots for consistency). Also make sure there is a valid vCenter backup (VAMI based backup or 3rd party image based backup). 

1. connect to the vCenter using root via an SSH session
   
   
2. Stop services that are using the database to prevent possible conflicts or issue:
   
   service-control --stop content-library
service-control --stop eam
service-control --stop vpxd
3. Connect to the vCenter database (VCDB) using the following command:
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB


4. Check the VCDB for the old agent entries:
   
   select * from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:##########';
   
   
5. Once the agent entry on the database has been verified, remove entries with the following command :
   
   delete from VPX_EXT_DATA where EXT_ID = 'com.vmware.vim.eam' and data_key = 'Agency:##########';