Trying to remove a datastore cluster fails and causes vCenter to crash

Products

• VMware vCenter Server

Issue/Introduction

Issue:

Unable to delete a datastore cluster in vCenter with the vSphere Web Client, no tasks are created, and each attempt causes vCenter to crash.

 

Using PowerCLI to try remove it shows the following:

Remove-DatastoreCluster : Remove-DatastoreCluster         The HTTP service located at https://######/sdk is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later.
At line:1 char:45
+ Get-DatastoreCluster -Name '######' | Remove-DatastoreCluster
+                                             ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Remove-DatastoreCluster], ViError
    + FullyQualifiedErrorId : Client20_StorageServiceImpl_RemoveDatastoreCluster_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.RemoveDatastoreCluster

 

Environment

vCenter Server 7.0.3

Cause

In the vCenter vpxd logs (/var/log/vmware/vpxd/vpxd.log) you see the following:

error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] Execute result code: -1
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] SQL execution failed: DELETE FROM VPX_ENTITY WHERE ID=?
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] Execution elapsed time: 5 ms
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] Statement diagnostic data from driver is 23503:0:1:ERROR: update or delete on table "vpx_entity" violates foreign key constraint "fk_vpx_enti_ref_vpx_enti" on table "vpx_entity"
--> DETAIL: Key (id)=(216145) is still referenced from table "vpx_entity".;
--> Error while executing the query
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] Bind parameters:
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] [0]datatype: 3, size: 8, arraySize: 0
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] value = 216145
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [Vdb::IsRecoverableErrorCode] Unable to recover from 23503:1
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] [VdbStatement] SQLError was thrown: "ODBC error: (23503) - ERROR: update or delete on table "vpx_entity" violates foreign key constraint "fk_vpx_enti_ref_vpx_enti" on table "vpx_entity"
--> DETAIL: Key (id)=(216145) is still referenced from table "vpx_entity".;
--> Error while executing the query" is returned when executing SQL statement "DELETE FROM VPX_ENTITY WHERE ID=?"
panic vpxd[07274] [Originator@6876 sub=vpxCommon opID=333f7014] Unrecoverable VmRootError: 00007f0bf03fefa0 Backtrace: 
--> Backtrace:
...
--> 
error vpxd[07274] [Originator@6876 sub=Default opID=333f7014] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23503) - ERROR: update or delete on table "vpx_entity" violates foreign key constraint "fk_vpx_enti_ref_vpx_enti" on table "vpx_entity"
--> DETAIL: Key (id)=(216145) is still referenced from table "vpx_entity".;
--> Error while executing the query" is returned when executing SQL statement "DELETE FROM VPX_ENTITY WHERE ID=?"
panic vpxd[07274] [Originator@6876 sub=Default opID=333f7014] 
--> 
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
...
--> [backtrace end]

Resolution

Resolution involves accessing the vCenter Database and deleting the entry. Open a ticket with support to verify the issue and obtain the resolution steps.

Interacting with the vCenter Server Appliance 6.5/6.7/7.0/8.0 embedded vPostgres Database
https://knowledge.broadcom.com/external/article/313597/interacting-with-the-vcenter-server-appl.html

• The article provide steps to connect to the embedded vPostgres Database of a vCenter Server Appliance and check the size of the tables.
• Caution: Ensure you have working backups and/or snapshots of the vCenter Server Appliance before interacting with the vCenter Server Appliance 6.5/6.7/7.0/8.0 Database.

 

Prior to proceeding with any changes, make sure to have powered off snapshots of the vCenter VM and all vCenter VMs if ELM is used. See KB 318333 below.

Snapshot Best practices for vCenter Server Virtual Machines
https://knowledge.broadcom.com/external/article/318333/snapshot-best-practices-for-vcenter-serv.html