vCenter Server 6.5 crashes with error: duplicate key value violates unique constraint "pk_vpx_guest_disk"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vpxd process crashes randomly after upgrading to vCenter Server 6.5 with error:

ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_guest_disk";
Panic: Unrecoverable database error. Shutting down VC

• In the vpxd.log file, you see entries similar to:

error vpxd[7F8DD228C700] [Originator@6876 sub=InvtVmDb opID=HB-host-476@72123-38e1cc31] >[VpxdInvtVm::SaveGuestNetworkAndDiskToDb] Failed to insert guest disk info for VM id = 976 because of database error: "ODBC error: >(23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_guest_disk";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_GUEST_DISK (VM_ID, PATH, CAPACITY, >FREE_SPACE) VALUES (?, ?, ?, ?)"

 

Note: vpxd.log file location:
For vCenter Server: %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\vmware-vpx

For vCenter Server Appliance: /var/log/vmware/vmware-vpx

• In the postgresql.log file, you see entries similar to:

VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_guest_disk"
VCDB vc DETAIL: Key (vm_id, path)=(976, /tmp) already exists.

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x

Resolution

This issue is resolved in vCenter Server 6.5.0e, available at Broadcom Support Portal.

Workaround:

vCenter Services observed to be returning to functional state with a service start. 

• How to Stop, Start or Restart vCenter Server 6.x Services

If starting failed to initialize the service with the same crash reason, workaround the condition by removing the impacted guest disk entry from vCenter Server Database. This information is safe to remove as it will be re-populated from host.

For vCenter Server with vPostgres database:

Note: Take a snapshot of the vCenter Sever machine before proceeding.

Connect the vCenter Database. For more information: 

1. • Interacting with the vCenter Server Appliance 6.5/6.7/7.0 embedded vPostgres Database
2. Identify the guest disk entry using below query:

select FROM vc.vpx_guest_disk where vm_id=<vm id> and path='/tmp';

For example:

select FROM vc.vpx_guest_disk where vm_id='976' and path='/tmp';

Note: The vm id is available from the vpxd.log error entry

3. To delete the duplicate value :-

delete FROM vc.vpx_guest_disk where vm_id=<vm id> and path='/tmp';
 

For example:

select FROM vc.vpx_guest_disk where vm_id='976' and path='/tmp';

Note: The vm id is available from the vpxd.log error entry

4. Start the Service. 
   • How to Stop, Start or Restart vCenter Server 6.x Services

Note: Remember to  delete the snapshot after observing stability of vCenter Server