vCenter vpxd service panics with error: "Double register of key"

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server UI is inaccessible, displaying the error "no healthy upstream".
• The vmware-vpxd service fails to start or crashes immediately upon service restart.
• Below entries are observed in /var/log/vmware/vpxd/vpxd.log:

  ERROR vpxd 3####61 [vc sub="Vmomi"] [VpxVmomi] Double register of key: 'vm-1234' and name: ''
  CRITICAL vpxd 3####61 [vc sub="Default"]
  -->
  --> Panic: NOT_REACHED bora/vpx/common/vpxVmomi.cpp:1162
  -->
  --> Backtrace:
  --> [backtrace begin] product: VMware VirtualCenter, version: 9.1.0, build: build-25370922, tag: vpxd, cpu: x86_64, os: linux, buildType: release
  --> backtrace[00] libvmacore.so[0x004###64]
  --> backtrace[01] libvmacore.so[0x003###AC]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
  --> backtrace[02] libvmacore.so[0x0037###9]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
  --> backtrace[03] libvmacore.so[0x0####264]
  --> backtrace[04] libvmacore.so[0x00####F8]: Vmacore::PanicExit(char const*)
  --> backtrace[05] libvmacore.so[0x004###5A]
  --> backtrace[06] libvmacore.so[0x0040###8]
  --> backtrace[07] vpxd[0x02###0E]
  --> backtrace[08] vpxd[0x015###9]
  --> backtrace[09] vpxd[0x013###E1]
  --> backtrace[10] vpxd[0x013###A4]
  --> backtrace[11] vpxd[0x013###EF]
  --> backtrace[12] libc.so.6[0x00####4A]
  --> backtrace[13] libc.so.6[0x000###05]
  --> backtrace[14] vpxd[0x013D###9]
  --> backtrace[15] (no module)
  --> [backtrace end]

Environment

VMware vCenter Server 9.x

Cause

A storage outage caused the vpxd service to crash, leading to unrecoverable vpxd database corruption. Database queries confirm no duplicate host entries exist in the vpx_host table, isolating the root cause to generalized database corruption resulting from the storage failure.

Resolution

1. Take the SSH session to vCenter using root credentials.
2. Check the vpxd.log to identify the specific VM key causing the conflict (e.g., vm-1234).
3. Stop the vpxd service:

    service-control --stop vmware-vpxd

4. Connect to the VCDB using psql:

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

5. Locate and remove the duplicate entries ( e.g., vm-1234 )  if found from the inventory tables. Please refer: VPXD crashes due to ERROR: duplicate key value
6.  If manual database modifications fail to resolve the issue due to widespread corruption, a restore of the vCenter Server is required. Refer: Restore vCenter from a File-Based Backup