vpxd crash with error "duplicate key value violates unique constraint "pk_vpx_entity"" after reverting vCenter Server to a snapshot

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• vCenter Server has been reverted to a snapshot
• The vCenter Server version when reverted is earlier than 8.0U2
• Checking the /var/log/vmware/vpxd/vpxd.log you see entries similar to the following leading up to the crash:

2023-10-09T13:28:27.459+01:00 error vpxd[101421] [Originator@6876 sub=Default opID=dasConfig.cpp:407-5f84f69b-cc61445-64aec1ee] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_entity"
--> DETAIL: Key (id)=(123456) already exists.;
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_ENTITY (ID,NAME,TYPE_ID,PARENT_ID) VALUES (?,?,?,?)"
2023-10-09T13:28:27.469+01:00 panic vpxd[101421] [Originator@6876 sub=Default opID=dasConfig.cpp:407-5f84f69b-cc61445-64aec1ee]
-->
--> Panic: Unrecoverable VmRootError. Panic!

Environment

VMware vCenter Server 8.0.0
VMware vCenter Server 8.0.2
VMware vCenter Server 8.0.1

Cause

When the vCenter Server is reverted to a snapshot, the value of sequence for object IDs is also reverted to the previous state. If there were Virtual Machines created after the snapshot was taken the possibility arises that vCenter Server could attempt to use the same object ID again. This causes a clash of object IDs with the value persisted in the Virtual Machines vmx file which results in the above duplicate key error.

Resolution

This issue is resolved in below mentioned versions:

VMware vCenter Server 8.0U2

VMware vCenter Server 7.0 Update 3q

Workaround:

To workaround this issue, run the following command on the vCenter Server which should allow vpxd to start:
/usr/lib/vmware-vpx/py/xmlcfg.py -f /etc/vmware-vpx/vpxd.cfg set vpxd/entityStoreRecovery/ongoingEnabled false