VPXD crashes due to ERROR: duplicate key value violates unique constraint "pk_vm_text"

Products

• VMware vCenter Server

Issue/Introduction

The vCenter UI is inaccessible and the vpxd service repeatedly crashes.

/var/log/vmware/vpxd/vpxd.log entries:

yyyy-mm-dd error vpxd[####] [Originator@#### sub=Default opID=HB-host-#####@#####-#####] [VdbStatement] Statement diagnostic data from driver is ####:#:#:ERROR: duplicate key value violates unique constraint "pk_vm_text"
--> DETAIL: Key (id, type_id)=(123456) already exists.;
--> Error while executing the query

Caution: Please consult a Broadcom Support Engineer before performing these steps

 

Warnings:

Before performing any database operations, take a full backup of the database or of the vCenter Server. VMware recommends that a consultation with a Database Administrator (DBA) before implementing any changes to the database configuration.

Environment

vCenter Server 7.0U3

Cause

A duplicate value in the VCDB causes the VPXD service to crash every minute.

Resolution

NOTE: Make sure all vCenters in ELM are shut down and take a snapshot of all nodes for backup. For standalone vCenters, a powered-on snapshot will be sufficient. Please check the below article for snapshot best practices:

• Best practices for using VMware snapshots in the vSphere environment
• Snapshot Best practices for vCenter Server Virtual Machines
• VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

After identifying the impacted VM_ID value from the vpxd logs. We need to follow the below steps to resolve this issue.

1. SSH to the vCenter server via root.

2. Stop the VPXD service, if the service has not yet crashed in your context.

# service-control --stop vpxd

3. Connect to the VCDB:

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

4. Running the following command lets us identify the VM associated with the VM_ID found in the vpxd.log

VCDB=# select id, file_name from vpx_vm where id = 123456;

   id    |                                             file_name
---------+----------------------------------------------------------------------------------------------------
 123456 | ds:///vmfs/volumes/Datastore-UUID/VM_name/VM_name.vmx
(1 row)

5. Delete the duplicate entries from VCDB by running the below commands for the corresponding VM_ID:

delete from  VPX_COMPUTE_RESOURCE_DAS_VM where VM_ID=123456;
delete from  VPX_COMPUTE_RESOURCE_DRS_VM where VM_ID=123456;
delete from  VPX_COMPUTE_RESOURCE_ORC_VM where VM_ID=123456;
delete from  VPX_VM_SGXINFO where VM_ID=123456;
delete from  VPX_GUEST_DISK where VM_ID=123456;
delete from  VPX_VM_VIRTUAL_DEVICE where ID=123456;
delete from  VPX_VM_DS_SPACE where VM_ID=123456;
delete from  VPX_NON_ORM_VM_CONFIG_INFO where ID=123456;
delete from  VPX_NORM_VM_FLE_FILE_INFO where VM_ID=123456;
delete from  VPX_VDEVICE_BACKING_REL where VM_ID=123456;
delete from  VPX_VIRTUAL_DISK_IOFILTERS where VM_ID=123456;
delete from  VPX_VM_STATIC_OVERHEAD_MAP where VM_ID=123456;
delete from  VPX_VM_TEXT where VM_ID=123456;
delete from  VPX_VM where ID=123456;
delete from  VPX_ENTITY where ID=123456;

6. Disconnect from the VCDB.

\q 

7. Start the VPXD service:

# service-control --start vpxd

Note: If the above command fails to start vpxd, restart all services instead

# service-control --stop --all; service-control --start --all

8. Monitor the start of the VPXD service or all services in a separate SSH instance.

# watch service-control --status --all