Patching vCenter 8.x fails with an error "Database in-place upgrade failed due to "RESTART value (XXXXXXXXXX) cannot be greater than MAXVALUE (4294965000)"

Products

• VMware vCenter Server

Issue/Introduction

• Patching VCSA 8.x using VAMI fails with the following error message:

Database in-place upgrade failed. Please see vcdb_inplace.err and vcdb_inplace.out for details.

• /var/log/vmware/applmgmt/Patchrunner.log contains the following entries:

   XXXX-XX-XXTXX:XX:XX.376Z psql.bin:/storage/seat/software-update9ffo1yec/stage/scripts/patches/payload/components-script/vcdb/ndu/vc_sql_procs/vcdb_db_upgrade.sql:176: ERROR:  22023,RESTART value (XXXXXXXXXX) cannot be greater than MAXVALUE (4294965000)
  CONTEXT:  PL/pgSQL function vcdb_ndu_cln_processor(text[]) line 20 at RAISE
  XXXX-XX-XXTXX:XX:XX.371Z vcdb:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vcdb:Patch' failed.
  Traceback (most recent call last):
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
      executionResult = systemExtension(args)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
      result = self.extension(*args)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
      return func(*args)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/payload/components-script/vcdb/__init__.py", line 192, in patch
      _update_db()
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/payload/components-script/vcdb/__init__.py", line 110, in _update_db
      raise UserError(cause=cause)
  patch_errors.UserError: Failed to update DB.
  XXXX-XX-XXTXX:XX:XX.376Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
  Traceback (most recent call last):
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
      _patchComponents(ctx, userData, statusAggregator.reportingQueue)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents
      executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
      reportQueue, identifier, expectedResultType)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
      result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
    File "/storage/seat/software-update9ffo1yec/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
      raise ex
  patch_errors.ComponentError
  XXXX-XX-XXTXX:XX:XX.378Z WARNING root stopping status aggregation...
  XXXX-XX-XXTXX:XX:XX.380Z ERROR __main__ Patch vCSA failed

Environment

VMware vCenter Server 8.0.x

Cause

This happens when vpx_event_seq contains values that are too high and needs to be reset.

Resolution

Before attempting the following steps, revert to the last backup or snapshot taken prior to the failed patch attempt.

Once this was done, please make sure to create a fresh offline snapshot of the vCenter Server Appliance (VCSA) in powered off state. If the affected VCSA is part of an Enhanced Linked Mode replication setup, please be aware that offline snapshots of all other ELM nodes are required as well.

To fix the issue:

1. Stop the vCenter Server service:

   # service-control --stop vpxd

2. Run the following command to regenerate vpx_event_seq:

   # /opt/vmware/vpostgres/current/bin/psql -U postgres VCDB -c "ALTER SEQUENCE VPX_EVENT_SEQ RESTART WITH 1" 

3. Clean-up the event tables in the vCenter DB by following the steps outlined in Delete old tasks, events and statistics data in vCenter Server 5.x, 6.x, 7.x and 8.x
4. Restart the vCenter Server service:

   # service-control --stop vpxd

5. Initiate the update again