vpxd crashes and reports error: "Double register of key: 'vm-xxxxx' and name: '<VM NAME>'" after restoring vCenter Server

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• vpxd crashes after restoring vCenter Server and creating VM.
• In the vpxd.log file, you see log entries similar to:

YYYY-MM-DDThh:mm:ss error vpxd[xxxxx] [Originator@xxxx sub=Vmomi opID=xxxxxxxx-xx] [VpxVmomi] Double register of key: 'vm-xxxxxx' and name: '<VM NAME>'
YYYY-MM-DDThh:mm:ss panic vpxd[xxxxx] [Originator@xxxx sub=Default opID=xxxxxxxx-xx]
-->
--> Panic: NOT_REACHED bora/vpx/common/vpxVmomi.cpp:###
-->
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: #.#.#, build: build-########, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0037DA99]
--> backtrace[01] libvmacore.so[0x002C78D3]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002D6B69]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x00370BE5]
--> backtrace[04] libvmacore.so[0x00370D01]: Vmacore::PanicExit(char const*)
--> backtrace[05] libvmacore.so[0x0033614B]
--> backtrace[06] libvmacore.so[0x003361E9]
--> backtrace[07] vpxd[0x01619AA3]
--> backtrace[08] vpxd[0x0091E001]
--> backtrace[09] vpxd[0x009013DF]
--> backtrace[10] vpxd[0x00902509]
--> backtrace[11] vpxd[0x012E9403]
--> backtrace[12] vpxd[0x012E96F5]
--> backtrace[13] vpxd[0x012D3842]
--> backtrace[14] vpxd[0x012C8C7C]
--> backtrace[15] vpxd[0x012CAACD]
--> backtrace[16] vpxd[0x012CB0EB]
--> backtrace[17] vpxd[0x0160D9BD]
--> backtrace[18] vpxd[0x0160DAFE]
--> backtrace[19] vpxd[0x0160EA2C]
--> backtrace[20] vpxd[0x0160178F]
--> backtrace[21] vpxd[0x0160705E]
--> backtrace[22] vpxd[0x012AE097]
--> backtrace[23] vpxd[0x012AE1C7]
--> backtrace[24] vpxd[0x008BC728]
--> backtrace[25] libvim-types.so[0x0105DF36]
--> backtrace[26] vpxd[0x0161D6C2]
--> backtrace[27] vpxd[0x0160D9BD]
--> backtrace[28] vpxd[0x0160DAFE]
--> backtrace[29] vpxd[0x0160EA2C]
--> backtrace[30] vpxd[0x0160178F]
--> backtrace[31] vpxd[0x0160C15A]
--> backtrace[32] libvmacore.so[0x002349A7]
--> backtrace[33] libvmacore.so[0x00239F4F]

• If vCenter HA is enabled, failover occurs because of vpxd crash

Environment

vCenter Server 7.x, vCenter Server 8.x

Cause

vCenter Server manages vmids, assigning them incrementally to virtual machines.
This ID is also recorded within each virtual machine's configuration file.
When vCenter Server is restored from a backup, the next vmid to be assigned also reverts to the state at the time the backup was taken.
Therefore, if any virtual machines were created between taking the vCenter Server backup and performing the restore, the vmid might conflict with newly assigned vmid after the restore, leading to a vpxd crash.

Example:
VM A is created, vCenter Server assigns vmid-10001.
-> vCenter Server backup is created.
-> VM B is created, vCenter Server assigns vmid-10002.
-> vCenter Server is restored from the backup.
-> VM C is created, and vCenter Server attempts to assign vmid-10002.
-> The IDs of VM B and VM C conflict, causing vpxd to crash.

Resolution

This issue is resolved in vCenter Server 7.0 U3q and 8.0 U2.