vpxd service crashes with error "duplicate key value violates unique constraint "pk_vpx_entity"" after reverting vCenter Server to a snapshot

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• vCenter UI is inaccessible with an error "No healthy upstream"
• vCenter Server has been reverted to a snapshot
• The vCenter Server version, when reverted is earlier than vCenter Server version 8.0U2 
• Checking the /var/log/vmware/vpxd/vpxd.log, you see entries similar to the following leading up to the crash:

           error vpxd[######] [Originator@xxxx sub=Default opID=dasConfig.cpp:###-#####-#####-######] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (#####) - ERROR: duplicate keybvalue violates unique constraint "pk_vpx_entity"
      --> DETAIL: Key (id)=(#####) already exists.;
      --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_ENTITY (ID,NAME,TYPE_ID,PARENT_ID) VALUES (?,?,?,?)"
      panic vpxd[######] [Originator@6876 sub=Default opID=dasConfig.cpp:###-#####-#####-######]
      -->
      --> Panic: Unrecoverable VmRootError. Panic!

Environment

vCenter 8.x
vCenter 7.x

Cause

When the vCenter Server is reverted to a snapshot, the value of sequence for object IDs is also reverted to the previous state. If there were virtual machines created after the snapshot was taken, the possibility arises that vCenter Server could attempt to use the same object ID again. This causes a clash of object IDs with the value persisted in the virtual machines vmx file which results in the above duplicate key error.

Resolution

This issue is resolved in vCenter Server versions: 8.0U2 and 7.0U3q.

Log in to the Broadcom Support Portal to download this patch: vCenter Server 8.0U2 and vCenter Server 7.0U3q.

To workaround this issue, run the following command on the vCenter Server, which should allow vpxd to start:

• /usr/lib/vmware-vpx/py/xmlcfg.py -f /etc/vmware-vpx/vpxd.cfg set vpxd/entityStoreRecovery/ongoingEnabled false