vpxd crashes with ERROR: duplicate key value violates unique constraint "pk_vpx_dvport

Products

VMware vCenter Server VMware vCenter Server 8.0 VMware NSX

Issue/Introduction

The vpxd service crashes after a short period of time running.
/storage/core has one or more files named similar to core.vpxd-worker.#######
vpxd.log shows the same or similar log entries:

####-##-##T##:##:##.######:## error vpxd[77553] [Originator@6876 sub=Default opID=HB-host-######@######-########] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_dvport_membership"
--> DETAIL: Key (dvs_id, dvport_key)=(####, ########-####-####-####-############) already exists.;
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_DVPORT_MEMBERSHIP (DVS_ID,DVPORT_KEY,DVPORTGROUP_ID,HOST_ID) VALUES (?,?,?,?)"

####-##-##T##:##:##.######:## error vpxd[77553] [Originator@6876 sub=Default opID=HB-host-######@######-########] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_dvport_membership"
--> DETAIL: Key (dvs_id, dvport_key)=(####, ########-####-####-####-############) already exists.;
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_DVPORT_MEMBERSHIP (DVS_ID,DVPORT_KEY,DVPORTGROUP_ID,HOST_ID) VALUES (?,?,?,?)"
####-##-##T##:##:##.######:## panic vpxd[77553] [Originator@6876 sub=Default opID=HB-host-268377@266355-57f448d9]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 8.0.3, build: build-24674346, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00531DD5]
--> backtrace[01] libvmacore.so[0x00421834]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x00434013]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x0050A993]
--> backtrace[04] libvmacore.so[0x0050AAAB]: Vmacore::PanicExit(char const*)
--> backtrace[05] vpxd[0x02829C53]
--> backtrace[06] vpxd[0x0280D2C5]
--> backtrace[07] vpxd[0x0280E0FD]
--> backtrace[08] vpxd[0x027FF3F1]
--> backtrace[09] vpxd[0x0280CAE0]
--> backtrace[10] libvmacore.so[0x0037EC04]
--> backtrace[11] libvmacore.so[0x00384517]
--> backtrace[12] libvmacore.so[0x00510FC5]
--> backtrace[13] libpthread.so.0[0x00008EB0]
--> backtrace[14] libc.so.6[0x000FFADF]
--> backtrace[15] (no module)
--> [backtrace end]

Environment

VMware vCenter Server 8.x
VMware NSX 4.x

Cause

A data inconsistency occurs when NSX and vCenter exchange/sync information, and then that information is loaded into an in-memory cache.

When a portgroup is missing, the corresponding port entry in VPX_DVPORT_MEMBERSHIP is skipped and therefore never loaded into cache. In some cases, portgroups may be removed without cleaning up their associated dvports.

Because of this, DVports became orphaned DVports, which caused several inconsistent behaviours on the DVS side. However, vCenter does not fully resolve dependencies when an NSX DVPG is removed. NSX should normally prevent a DVPG from being deleted while it is still in use, but in some cases dvports are left behind.

Resolution

This is a known issue impacting VMware vCenter.
Subscribe to this KB article to be updated if a fix version becomes available.

Workaround:

Contact Broadcom Support for a workaround and immediate assistance.

Additional Information

vCenter Server Update and Patch Release Notes