Error: tuple concurrently deleted causing no healthy upstream after power shutdown

Products

• VMware vCenter Server

Issue/Introduction

• The vmware-vpxd service fails to start or crashes unexpectedly on a standalone VMware vCenter Server Appliance. This behavior is typically observed following an abrupt power shutdown or hardware failure. Review of the vpxd.log shows an unrecoverable ODBC error related to the vpx_vm table in the embedded vPostgres database.

   

• Users see a "no healthy upstream" error when accessing the vCenter Server UI.

• The vmware-vpxd service status shows as failed or stopped.

• In /var/log/vmware/vpxd/vpxd.log, you see entries similar to:

Error: Error[VdbODBCError] (-1) "ODBC error: (XX000) - ERROR: tuple concurrently deleted; -- > Error while executing the query" is returned when executing SQL statement "UPDATE VPX_VM SET DATACENTER_ID =? ... Panic: Unrecoverable VmRootError. Panic!

 

Environment

• VMware vCenter 8.x

Cause

• A concurrency conflict or transactional row corruption has occurred within the embedded vPostgres database inside the vpx_vm table. This occurs when a transactional thread attempts to update a row that was concurrently modified or deleted by a parallel database session, often exacerbated by an abrupt system shutdown.

Resolution

To resolve this issue, you must purge dead tuples and rebuild corrupted indexes on the target inventory table.

1. Before performing manual modifications to the vCenter Server database, perform a file-based backup or an offline consistent snapshot of the standalone vCenter Server Appliance.
2. Access the CLI: Log into the vCenter Server Appliance via SSH using root credentials.
3. Stop the VirtualCenter Service: Stop the vpxd management service to remove active write locks on the database:
   
   service-control --stop vmware-vpxd
4. Access vPostgres: Connect to the embedded database instance:.
   
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
5. Reclaim Space and Reindex: Run a full vacuum optimization and rebuild the table's index maps. Ensure the /storage/db partition has at least 50% available free space relative to the table size before executing:
   
   VACUUM FULL ANALYZE vpx_vm;
REINDEX TABLE vpx_vm;
\q
6. Restart Services: Start the VirtualCenter management stack:

service-control --start vmware-vpxd

If the issue persists, please Contact Broadcom Support.