VPXD stops immediately after it is started with the error ""ODBC error: (22003) - ERROR: integer out of range;"

Products

• VMware vCenter Server

Issue/Introduction

VPXD service crashes and crashes after subsequent start attempts.

vpxd.log

--> SymBacktrace[24] 00007fed2406da10 rip=00007fed2acc8f4f in function (null) in object /usr/lib/vmware-vpx/libvmacore.so loaded at 00007fed2aa8f000
--> SymBacktrace[25] 00007fed2406daa0 rip=00007fed2ae054ac in function (null) in object /usr/lib/vmware-vpx/libvmacore.so loaded at 00007fed2aa8f000
--> SymBacktrace[26] 00007fed2406dad0 rip=00007fed27455f87 in function (null) in object /lib/libpthread.so.0 loaded at 00007fed2744e000
--> SymBacktrace[27] 00007fed2406db80 rip=00007fed2738962f in function clone in object /lib/libc.so.6 loaded at 00007fed27296000
--> SymBacktrace[28] 00007fed2406db88 rip=0000000000000000
-->
2023-07-05T03:29:51.195Z error vpxd[32560] [Originator@6876 sub=Default opID=HB-host-165@582281-49ae0caf] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (22003) - ERROR: integer out of range;
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_DVS_BLOB (DVS_ID,DVPORTGROUP_ID,DVPORT_KEY,HOST_ID,PROPERTY_PATH,BLOB_KEY,BLOB_DATA) VALUES (?,?,?,?,?,?,?)"
2023-07-05T03:29:51.240Z panic vpxd[32560] [Originator@6876 sub=Default opID=HB-host-165@582281-49ae0caf]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-20990077, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0037DA77]
--> backtrace[01] libvmacore.so[0x002C78D3]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002D6B69]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&

 

 

Environment

VMware vCenter Server 7.0.x

Cause

The surr_key numbers of the table VPX_DVS_BLOB have hit the end of the defined range, thus any further numbers are considered as out of range and the table will not allow the field to be updated with the out-of-range numbers.

Resolution

1. Take a snapshot of the vCenter (if ELM, take offline snapshot of all vCenters in the ELM - VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice)
   
   
2. SSH to the VCSA as the root user
   
   
   
3. Connect to the vPostgres DB - Interacting with the vCenter Server Appliance 6.5/6.7/7.0/8.0 embedded vPostgres Database
   
   
   
4. Execute the following commands.  *Note we are using the example VPX_DVS_BLOB found from the vpxd.log
   
   do $$ declare ct tid; cnt bigint := 1; begin for ct in (select ctid from vc.vpx_dvs_blob order by surr_key) loop update vc.vpx_dvs_blob set surr_key = cnt where ctid = ct; cnt := cnt +1; end loop; end; $$; select setval(pg_get_serial_sequence ( 'vc.vpx_dvs_blob', 'surr_key' ), (select max(surr_key) from vc.vpx_dvs_blob) + 100) ;
   
   
5. Quit vPostgres
   
   \q


6. Start VPXD 
   
   service-control --start vpxd

 

 

Additional Information

Impact/Risks:

VPXD will stop immediately after it starts and causes vCenter unusable.