vpxd crashes when changing vSphere Distributed Switch (vDS) settings when using postgres database

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter Server Appliance (vpxd) crashes when changing vDS settings.
The vpxd.log file shows errors similar to:

YYYY-MM-DDTHH:MM:SS.MS error vpxd[7FC8EEA3E700] [Originator@6876 sub=Default opID=DvPortSettingsDialog-apply-1549-ngc:70000253-b2] [Vdb::IsRecoverableErrorCode] Unable to recover from 23505:7
YYYY-MM-DDTHH:MM:SS.MS error vpxd[7FC8EEA3E700] [Originator@6876 sub=Default opID=DvPortSettingsDialog-apply-1549-ngc:70000253-b2] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110
YYYY-MM-DDTHH:MM:SS.MS error vpxd[7FC8EEA3E700] [Originator@6876 sub=vpxCommon opID=DvPortSettingsDialog-apply-1549-ngc:70000253-b2] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
YYYY-MM-DDTHH:MM:SS.MS error vpxd[7FC8EEA3E700] [Originator@6876 sub=Default opID=DvPortSettingsDialog-apply-1549-ngc:70000253-b2] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbError] (-17) "Error for Update" is returned when executing SQL statement "UPDATE VPX_DVPORT_SETTING SET BLOCKED_FLG = ? , NRP_KEY = ? , IN_SHAPING_OBJECT_FLG = ? , IN_SHAPING_AVG_BANDWIDTH = ? , IN_SHAPING_BURST_SIZE = ? , IN_SHAPING_ENABLED = ? , IN_SHAPING_PEAK_BANDWIDTH = ? , OUT_SHAPING_OBJECT_FLG = ? , OUT_SHAPING_AVG_BANDWIDTH = ? , OUT_SHAPING_BURST_SIZE = ? , OUT_SHAPING_ENABLED = ? , OUT_SHAPING_PEAK_BANDWIDTH = ? , VLAN_OBJECT_FLG = ? , VLAN_ID = ? , QOS_TAG = ? , TX_UPLINK = ? , IPFIX_OBJECT_FLG = ? , IPFIX_ENABLED = ? , SEC_POLICY_OBJECT_FLG = ? , ALLOW_PROMISCUOUS = ? , FORGED_TRANSMITS = ? , MAC_CHANGES = ? , TEAMING_OBJECT_FLG = ? , TEAMING_POLICY = ? , TEAMING_REVERSE = ? , TEAMING_NOTIFY = ? , TEAMING_ROLLING = ? , CRITERIA_OBJECT_FLG = ? , TEAMING_FAIL_CHK_BEACON = ? , TEAMING_FAIL_CHK_DUPLEX = ? , TEAMING_FAIL_CHK_PERCENT = ? , TEAMING_FAIL_CHK_SPEED = ? , TEAMING_FAIL_DUPLEX = ? , TEAMING_FAIL_PERCENT = ? , TEAMING_FAIL_SPEED = ? , UPLINK_ORDER_OBJECT_FLG = ? , VENDOR_BLOB_OBJECT_FLG = ? , NPT_ALLOWED_FLG = ? , LACP_ENABLE = ? , LACP_MODE = ? , PVLAN_ID = ? , OVERLAY_OBJECT_FLG = ? , TRAFFIC_FILTER_OBJ_FLG = ? WHERE DVS_ID = ? AND DVPORTGROUP_ID = ? AND DVPORT_KEY = ?"
YYYY-MM-DDTHH:MM:SS.MS panic vpxd[7FC8EEA3E700] [Originator@6876 sub=Default opID=DvPortSettingsDialog-apply-1549-ngc:70000253-b2]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-8307201, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B673C]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1FE8]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002B46EB]
--> backtrace[03] libvmacore.so[0x002B47C6]: Vmacore::PanicExit(char const*)
--> backtrace[04] vpxd[0x00726D5A]
--> backtrace[05] vpxd[0x0073C70B]
--> backtrace[06] vpxd[0x0073EA41]
--> backtrace[07] vpxd[0x00743D82]
--> backtrace[08] libvmacore.so[0x0023B4A3]
--> backtrace[09] libvmacore.so[0x0023B78A]
--> backtrace[10] libvmacore.so[0x002BBFDB]
--> backtrace[11] libpthread.so.0[0x00007444]
--> backtrace[12] libc.so.6[0x000E8C2D]
--> [backtrace end]

The vpxd.log also shows errors as similar to:

YYYY-MM-DDTHH:MM:SS.MS UTC 5b05b6ad.294b 0   LOG:  Updated instance status successfully.
YYYY-MM-DDTHH:MM:SS.MS UTC 5b05b6ed.32b5 669655 VCDB vc ERROR:  duplicate key value violates unique constraint "pk_vpx_dvport_setting"
YYYY-MM-DDTHH:MM:SS.MS UTC 5b05b6ed.32b5 669655 VCDB vc DETAIL:  Key (dvs_id, surr_key)=(21443, 14) already exists.
YYYY-MM-DDTHH:MM:SS.MS UTC 5b05b6ed.32b5 669655 VCDB vc STATEMENT:  INSERT INTO VPX_DVPORT_SETTING (DVS_ID, DVPORTGROUP_ID, DVPORT_KEY, BLOCKED_FLG, NRP_KEY, IN_SHAPING_OBJECT_FLG, IN_SHAPING_AVG_BANDWIDTH, IN_SHAPING_BURST_SIZE, IN_SHAPING_ENABLED, IN_SHAPING_PEAK_BANDWIDTH, OUT_SHAPING_OBJECT_FLG, OUT_SHAPING_AVG_BANDWIDTH, OUT_SHAPING_BURST_SIZE, OUT_SHAPING_ENABLED, OUT_SHAPING_PEAK_BANDWIDTH, VLAN_OBJECT_FLG, VLAN_ID, QOS_TAG, TX_UPLINK, IPFIX_OBJECT_FLG, IPFIX_ENABLED, SEC_POLICY_OBJECT_FLG, ALLOW_PROMISCUOUS, FORGED_TRANSMITS, MAC_CHANGES, TEAMING_OBJECT_FLG, TEAMING_POLICY, TEAMING_REVERSE, TEAMING_NOTIFY, TEAMING_ROLLING, CRITERIA_OBJECT_FLG, TEAMING_FAIL_CHK_BEACON, TEAMING_FAIL_CHK_DUPLEX, TEAMING_FAIL_CHK_PERCENT, TEAMING_FAIL_CHK_SPEED, TEAMING_FAIL_DUPLEX, TEAMING_FAIL_PERCENT, TEAMING_FAIL_SPEED, UPLINK_ORDER_OBJECT_FLG, VENDOR_BLOB_OBJECT_FLG, NPT_ALLOWED_FLG, LACP_ENABLE, LACP_MODE, PVLAN_ID, OVERLAY_OBJECT_FLG, TRAFFIC_FILTER_OBJ_FLG) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46)
YYYY-MM-DDTHH:MM:SS.MS UTC 5b05b6ad.294b 0   LOG:  Updating instance status...

Environment

VMware vCenter Server Appliance 6.5.x

Cause

Database sequence is out of sync during an attempt to either insert/update the primary key column on the VPX_DVPORT_SETTING table, causing the vCenter Server Appliance to crash.

Resolution

Note: This procedure modifies the database. Ensure to take a backup of the vCenter before proceeding.

Take snapshot of VC, if VC is in ELM make sure to take offline snapshot of all VCs.
SSH to vCenter using root credentials.
Please reach out Broadcom support if assistance needed while preforming the steps.

Update the last_value in the sequence to reflect the latest sequence +1.

Log into the vCenter Server Appliance as root via SSH or Console.
Run the following command to enter the shell.

shell

Connect to the vPostgres database for SQL execution by running this command:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

To get the current sequence last_value, run this command:

select * from vpx_dvport_setting_surr_key_seq;

Example: This will give 11 as the last value as in example below.

sequence_name | last_value | start_value | increment_by | max_value | min_value | cache_value | log_cnt | is_cycled | is_called
---------------------------------+------------+-------------+--------------+---------------------+-----------+-------------+---------+-----------+-----------
vpx_dvport_setting_surr_key_seq | 11 | 1 | 1 | 9223372036854775807 | 1 | 1 | 0 | f | t
(1 row)

To fix and reset the sequence value, run the statement below:

select setval('vpx_dvport_setting_surr_key_seq', (select coalesce(max(surr_key)+1) from vpx_dvport_setting), false);

To re check the sequence value, run the statement below:

select * from vpx_dvport_setting_surr_key_seq;

To re confirm the value of surr_key (seq value) run the statement:

select coalesce(max(surr_key)+1) from vpx_dvport_setting;