Unable to delete an empty cluster from the vCenter Inventory

Products

VMware VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

vCenter Server 6.x creates core.vpxd-worker files when trying to delete an empty cluster.
backtraces similar to the following in /var/log/vmware/vpxd/vpxd.log

The remove cluster task begins:

2021-02-03T21:17:27.154Z info vpxd[04533] [Originator@6876 sub=vpxLro opID=kkpxhkhp-536-auto-ex-h5:70000213-64] [VpxLRO] -- BEGIN task-381239 -- domain-c155012 -- vim.ManagedEntity.destroy -- 529235ce-0006-ea90-d9ff-2e1c8682cf17(5297fd20-863a-61ae-e0c3-7333b88942ca)

The task fails 30 seconds later with an EAM error:

2021-02-03T21:17:57.276Z error vpxd[04533] [Originator@6876 sub=Iofilter opID=kkpxhkhp-536-auto-ex-h5:70000213-64] Received EAM runtime fault: N3Eam5Fault10EamIOFault9ExceptionE(Fault cause: eam.fault.EamIOFault
2021-02-03T21:17:57.281Z warning vpxd[04533] [Originator@6876 sub=Iofilter opID=kkpxhkhp-536-auto-ex-h5:70000213-64] Failed to get vib information.Got exception: N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState

EAM logs state that its timing out trying to download a zip bundle (saocache-offline-bundle.zip) from a provider.

2021-02-03T21:27:29.636Z | ERROR | integrity-LRO | AuditedJob.java | 58 | JOB FAILED: [#338140271] com.vmware.eam.job.UrlHostJob@14279c6f
com.vmware.eam.EamIOException: URL: http://<PROVIDER-IP>:80/pkg/4.1.0.765/esxi6/4.1.0.64621/saocache-offline-bundle.zip can't be downloaded
        at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:80)
        at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:22)
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:48)
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out

...snip...

New issues:
 [
eam.issue.integrity.agency.CannotStageSoftware {
   time = 2021-02-03 21:27:29,638,
   description = 'Could not upload the following VIBs in Integrity server: [http://<PROVIDER-IP>:80/pkg/4.1.0.765/esxi6/4.1.0.64621/saocache-offline-bundle.zip]',
   key = 4,
   agency = 'Agency:55559444-1234-430d-1234-89c8c139eaf6:null',
   solutionId = 'VirtualCenter',
   agencyName = 'IoFilter-SAO_bootbank_saocache',
   solutionName = 'VirtualCenter',

Details above vary for different environments

Environment

VMware vSphere ESXi 6.x
VMware vCenter Server 6.x

Cause

This is generally observed in environments where previously used IO Filters were removed. IO Filters are third-party and in order to remove them, vCenter relies on their software.

Sometimes, this does not remove the IO Filter entries from VCDB, hence EAM will keep trying to reach out to the URL.

Resolution

To resolve this issue, we need to remove the IOFilter entry associated with the agency from the VPX_CLUSTER_IOFILTERS table in the VCDB

Ensure you have working backups and/or powered off snapshots of the vCenter (or ALL vCenters in case of ELM)
Connect to the VCDB

/opt/vmware/vpostgres/current/bin/psql -d VCDB postgres

View the contents of the VPX_CLUSTER_IOFILTERS table and find the row that matches the EAM log messages

select * from VPX_CLUSTER_IOFILTERS;

Delete the offending row from the table

delete from VPX_CLUSTER_IOFILTERS where cluster_id="CLUSTER-ID-FROM-STEP-3";