vCenter Fails to Start with the Error: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
search cancel

vCenter Fails to Start with the Error: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";

book

Article ID: 326594

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:
  • vCenter Server service VPXD crashes or fails to start.  In the log /var/log/vmware/vpxd/vpxd.log you see errors similar to:
error vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb] [VdbStatement] SQLError was thrown: "ODBC error: (23503) - ERROR: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_SN_CONFIG_INFO (ID, CHANGE_VERSION, ARRAY_ID, CHANGE_TRACKING_ENABLED, CPU_HOT_ADD_ENABLED, CPU_HOT_REMOVE_ENABLED, MEM_HOT_ADD_ENABLED, VASSERTS_ENABLED, HARDWARE_NUM, HARDWARE_MEMORY, TOOLS_BEFORE_GUEST_STANDBY_FLG, TOOLS_BEFORE_GUESTSHUTDOWN_FLG, TOOLS_TOOLS_UPGRADE_POLICY, TOOLS_AFTER_RESUME_FLG, TOOLS_AFTER_POWER_ON_FLG, TOOLS_SYNC_TIME_WITH_HOST_FLG, TOOLS_TOOLS_VERSION, GUEST_FULL_NAME, INSTANCE_UUID, UUID, ANNOTATION, VERSION, TEMPLATE_FLG, MEMORY_ALLOCATION_RESERVATION, MEMORY_ALLOC"
error vpxd[7F512A1C3700] [Originator@6876 sub=DbSave opID=HB-host-20@604097-691c56bb] [VpxdDbSave::ProcessInsertsInOrder] Failed to insert 2 records into the DB
error vpxd[7F512A1C3700] [Originator@6876 sub=vpxCommon opID=HB-host-20@604097-691c56bb] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
error vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23503) - ERROR: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_SN_CONFIG_INFO (ID, CHANGE_VERSION, ARRAY_ID, CHANGE_TRACKING_ENABLED, CPU_HOT_ADD_ENABLED, CPU_HOT_REMOVE_ENABLED, MEM_HOT_ADD_ENABLED, VASSERTS_ENABLED, HARDWARE_NUM, HARDWARE_MEMORY, TOOLS_BEFORE_GUEST_STANDBY_FLG, TOOLS_BEFORE_GUESTSHUTDOWN_FLG, TOOLS_TOOLS_UPGRADE_POLICY, TOOLS_AFTER_RESUME_FLG, TOOLS_AFTER_POWER_ON_FLG, TOOLS_SYNC_TIME_WITH_HOST_FLG, TOOLS_TOOLS_VERSION, GUEST_FULL_NAME, INSTANCE_UUID, UUID, ANNOTATION, VERSION, TEMPLATE_FLG, MEMORY_ALLOCATION_RESERVATION, MEMORY_ALLOC"
panic vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb]
--> 
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-7515524, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002B3E43]
--> backtrace[03] libvmacore.so[0x002B3F1E]: Vmacore::PanicExit(char const*)
--> backtrace[04] vpxd[0x0072584A]
--> backtrace[05] vpxd[0x0073B18B]
--> backtrace[06] vpxd[0x0073D47B]
--> backtrace[07] vpxd[0x007427B2]
--> backtrace[08] libvmacore.so[0x0023AC47]
--> backtrace[09] libvmacore.so[0x0023AF2E]
--> backtrace[10] libvmacore.so[0x002BB737]
--> backtrace[11] libpthread.so.0[0x00007454]
--> backtrace[12] libc.so.6[0x000E8C1D]
--> [backtrace end]


Environment

VMware vSAN 6.x

Cause

This issue is caused by a 256 character limit on the column DEVICE_INFO_SUMMARY in the tables VPX_VM_VIRTUAL_DEVICE and VPX_VM_SN_VIRTUAL_DEVICE.
This limit is hit when inserting information about a snapshot for a VM that resides on VSAN storage.

Resolution

This issue is resolved by increasing the length of the DEVICE_INFO_SUMMARY column in tables VPX_VM_VIRTUAL_DEVICE and VPX_VM_SN_VIRTUAL_DEVICE on the vCenter database.
  1. Log into the vCenter Service Appliance as root via SSH or Console.
  2. Shutdown Specific vCenter Server Services:
service-control --stop vmware-vpxd
service-control --stop vmware-content-library
  1. Connect to the vCenter Server PostgreSQL Database:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
  1. Run the following DDL commands on the vCenter Server PostgreSQL Database:
alter table VPX_VM_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY type varchar (512);
alter table VPX_VM_SN_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY type varchar (512);

For MSSQL Database:
alter table VPX_VM_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY nvarchar (512);
alter table VPX_VM_SN_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY nvarchar (512);

To exit the SQL DB run \q to quit
  1. Add the following to /etc/vmware-vpx/vpxd.cfg in the <config> section:
   <stressOption>
      <vdb>
         <schemaLoader>
             <forceODBC>1</forceODBC>
         </schemaLoader>
      </vdb>
   </stressOption>
  1. Start Specific vCenter Server Services
service-control --start vmware-vpxd
service-control --start vmware-content-library
  1. Ensure the changes to vpxd.cfg from Step 5 have been removed.