vCenter vmware-vpxd service crashes during cluster delete operation

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Deleting an empty vCenter cluster results in a vCenter vmware-vpxd service crash, and clusters cannot be deleted.
• /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS error vpxd[50997] [Originator@6876 sub=Default opID=1xxxxxxx-xxxxx-xxxxx0] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23503) - ERROR: update or delete on table "vpx_compute_resource" violates foreign key constraint "fk_vpx_comp_drs_vm_ref_vpx_com" on table "vpx_compute_resource_drs_vm"
--> DETAIL: Key (id)=(xxx) is still referenced from table "vpx_compute_resource_drs_vm".;
--> Error while executing the query" is returned when executing SQL statement "DELETE FROM VPX_ENTITY WHERE ID=?"
YYYY-MM-DDTHH:MM:SS panic vpxd[50997] [Originator@6876 sub=Default opID=1xxxxxxx-xxxxx-xxxxx0

--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-21784236, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0037DA99]
--> backtrace[01] libvmacore.so[0x002C78D3]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002D6B69]:         Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x00370BE5]
--> backtrace[04] libvmacore.so[0x00370D01]: Vmacore::PanicExit(char const*)
--> backtrace[05] vpxd[0x015E6495]
--> backtrace[06] vpxd[0x0160DC19]
--> backtrace[07] vpxd[0x0160EA2C]
--> backtrace[08] vpxd[0x0160178F]
--> backtrace[09] vpxd[0x0160C15A]
--> backtrace[10] libvmacore.so[0x002349A7]
--> backtrace[11] libvmacore.so[0x00239F4F]
--> backtrace[12] libvmacore.so[0x003764CE]
--> backtrace[13] libpthread.so.0[0x00007F87]
--> backtrace[14] libc.so.6[0x000F362F]
--> backtrace[15] (no module)
--> [backtrace end]

 

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

The vmware-vpxd service may crash or hang if a stale or duplicate entry exists for the empty clusters in vpx_compute_resource_drs_vm table in the VCDB.

Resolution

Before proceeding with the steps below, take both backup and a snapshot of the vCenter Server Appliance. If the vCenter is part of a Enhanced Linked Mode (ELM) replication setup, also take a backup or offline (powered off) snapshot of all replicating vCenter ELM nodes.

Recommendation: To resolve this issue, contact VMware by Broadcom Support. To contact VMware by Broadcom support, see Creating and managing Broadcom support cases

• SSH into the vCenter as the root user
• To find the affected cluster ID 
  • psql -h localhost -U postgres -d VCDB -c "select * from vpx_entity where name = '<affectedclustername>';"
• To find any stale drs rules related to that cluster
  • psql -h localhost -U postgres -d VCDB -c "select * from vpx_compute_resource_drs_vm where comp_res_id =  '<TheIdNumberFromAbove>';"
• To find the VM name of the vm ids listed
  • psql -h localhost -U postgres -d VCDB -c "select * from vpx_entity where id = 'TheVMIDFromAbove';"
• Unregister this VM from inventory.
• Delete the empty cluster.
• Re-register the VM.

Additional Information

If this doesn't resolve the issue please reach out to Broadcom support for assistance.