When any of the below actions are performed on an HCX IX appliance, the vCenter appliance VPXD service fails:
To resolve the issue related to VPXD crashed due to a duplicate value in VCDB.
Symptoms:
Stopped/Started all services, but some services did not started:Running:
applmgmt lookupsvc lwsmd observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
Stopped:
observability vmcam vmware-content-library vmware-imagebuilder vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp
Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting
VPXD service crashed with a backtrace.
vpxd.log:
[YYYY-MM-DDTHH:MM:SS] error vpxd[53839] [Originator@6876 sub=Default opID=HB-host-89@27928-513e6bca] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint
pk_vpx_nw_assignment"
-- > DETAIL: Key (network_id, entity_id)=(16001, 89) already exists .;
-- > Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_NW_ASSIGNMENT (NETWORK_ID,ENTITY_ID) VALUES (?,?)"
[YYYY-MM-DDTHH:MM:SS] panic vpxd[53839] [Originator@6876 sub=Default opID=HB-host-89@27928-513e6bca]
-- >
-- > Panic: Unrecoverable VmRootError. Panic!
-- > Backtrace:
-- > [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-22357613, tag: vpxd, cpu: x86_64, os: linux, buildType: release
backtrace[00] libvmacore.so[0x0037DA7F]
-- > backtrace[01] libvmacore.so[0x002C78B9]: Vmacore :: Stacktrace :: CaptureFullWork(unsigned int)
-- > backtrace[02] libvmacore.so[0x002D6B4F]: Vmacore :: SystemFactory :: CreateBacktrace(Vmacore :: Ref<Vmacore :: Backtrace>&)
-- > backtrace[03] libvmacore.so[0x00370BCB]
-- > backtrace[04] libvmacore.so[0x00370CE7]: Vmacore :: PanicExit(char const*)
-- > backtrace[05] vpxd[0x015E5697]
-- > backtrace[06] vpxd[0x0160CE45]
-- > backtrace[07] vpxd[0x0160DC58]
-- > backtrace[08] vpxd[0x016009BB]
-- > backtrace[09] vpxd[0x0160B386]
-- > backtrace[10] libvmacore.so[0x002349A7]
-- > backtrace[11] libvmacore.so[0x00239F35]
-- > backtrace[12] libvmacore.so[0x003764B4]
-- > backtrace[13] libpthread.so.0[0x00007F87]
-- > backtrace[14] libc.s0.6[0x000F362F]
-- > backtrace[15] (no module)
-- > [backtrace end]
VPXD service was unable to start due to duplicate key value that violates unique constraint "pk_vpx_nw_assignment"
Note: Ensure to take a backup of vpostgres database or snapshot of the appliance. For more information, see Back up and restore vCenter Server Appliance/vCenter Server 6.x vPostgres database
Workaround:
1. SSH to the VCSA via root.
2. Connect to the VCDB:/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
3. Run the following command to identify the VM ID associated with stated Key ID:
VCDB=# select * from vpx_nw_assignment where entity_id='89';
network_id | entity_id
------------+-----------
21 | 89
91 | 89
92 | 89
(3 rows)
VCDB=# select id,ip_address from vpx_host where id='89';
id | ip_address
----+---------------
89 | 10.219.49.139
(1 row)
VCDB=# select * from vpx_entity where id='89';
id | name | type_id | parent_id
----+---------------+---------+-----------
89 | 10.219.49.139 | 1 | 87
(1 row)
Note: IP: 10.219.49.139 >> HCX - IX appliance.
4. Disconnected the IX- appliance from the vcenter database and started the services of vcenter using below command:
VCDB=# update vpx_host set enabled=0 where id='89';
UPDATE 1
VCDB=# \q
5. Start all services: service-control --stop --all && service-control --start --all
6. vCenter came up and all the services started successfully.
Note:
After vpxd is online follow the workaround below to continue with the migration without impacting vpxd.