Upgrade or Patching to vCenter 7.0 Update 2 or later fails during vPostgres upgrade when amcheck_next extension is present in vCenter Database.
search cancel

Upgrade or Patching to vCenter 7.0 Update 2 or later fails during vPostgres upgrade when amcheck_next extension is present in vCenter Database.

book

Article ID: 336301

calendar_today

Updated On: 04-01-2025

Products

VMware vCenter Server

Issue/Introduction

  • Patching a vCenter 7.0 from 7.0 U1D or earlier to 7.0 U2 or later fails with below error. 
    Exception occured in postInstallHook

  • Error in /var/log/vmware/applmgmt/PatchRunner.log 
      Running: su -s /bin/bash vpostgres -c cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/12/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/12/bin -b /opt/vmware/vpostgres/12/../11/bin -D /storage/db/vpostgres.12 -d /storage/db/vpostgres --check
Running command: ['su', '-s', '/bin/bash', 'vpostgres', '-c', 'cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/12/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/12/bin -b /opt/vmware/vpostgres/12/../11/bin -D /storage/db/vpostgres.12 -d /storage/db/vpostgres --check']
  Done running command
, stderr:   pg_upgrade --check returned error code 1 with error  out Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Checking for presence of required libraries                 fatal

Your installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:
    loadable_libraries.txt

Failure, exiting

.
vpostgres:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpostgres:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/payload/components-script/vpostgres/__init__.py", line 189, in patch
    'PostgreSQL failed to run upgrade command.')))
patch_errors.UserError: PostgreSQL failed to run upgrade command.
 ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 199, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 86, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/seat/software-updatewpo_1lzb/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
 WARNING root stopping status aggregation...
 ERROR __main__ Patch vCSA failed
  • Checking the contents of loadable_libraries.txt shows a reference to "$libdir/amcheck_next" 
    $cat loadable_libraries.txt

could not load library "$libdir/amcheck_next": ERROR:  could not access file "$libdir/amcheck_next": No such file or directory
Database: VCDB

 

  • An Upgrade of vCenter from 6.x to 7.0 Update 2 or later fails during firstboot, with an error during vPostgres upgrade. In the location var/log/firstboot/firstbootInfrastructure.log we see the below: 
    firstbootInfrastructure.log :
firstbootInfrastructure [Failed] /usr/lib/vmware-vpostgres/firstboot/vpostgres-firstboot.py is complete
INFO firstbootInfrastructure No localized error detail found in /var/log/firstboot/vpostgres-firstboot.py_38958_stderr.log, assuming internal error
WARNING firstbootInfrastructure Bug component info file does not exist
INFO firstbootInfrastructure Firstboot duration: 68 sec
INFO firstbootInfrastructure First boot is a failure
ERROR firstbootInfrastructure Installation of vCenter server failed with firstboot scripts
WARNING firstbootInfrastructure stopping status aggregation...

vpostgres-firstboot.py_xxxxx_stderr.log :
Upgrade import step failed
vPostgres firstboot(action=firstboot) failed

vcdb_import.err :
psql.bin:/storage/seat/cis-export-folder/vcdb/VCDB_schema.dump:100: ERROR: could not open extension control file "/opt/vmware/vpostgres/12/share/extension/amcheck_next.control": No such file or directory
psql.bin:/storage/seat/cis-export-folder/vcdb/VCDB_schema.dump:107: ERROR: extension "amcheck_next" does not exist

postgresql-xx.log :
VCDB postgres ERROR: could not open extension control file "/opt/vmware/vpostgres/12/share/extension/amcheck_next.control": No such file or directory
VCDB postgres STATEMENT: CREATE EXTENSION IF NOT EXISTS amcheck_next WITH SCHEMA vc;
VCDB postgres ERROR: extension "amcheck_next" does not exist
VCDB postgres STATEMENT: COMMENT ON EXTENSION amcheck_next IS 'functions for verifying relation integrity';
LOG: Updating instance status...

 

  • On checking the vCenter Database of the vCenter being Updated or Upgraded there is an extension called "amcheck_next".




Environment

VMware vCenter Server 7.0.x

Cause

  • The vPostgres Consistency Check script vpgsql-consistency-check.sh creates the amcheck_next extension during the check.
  • Older versions of this script do not remove the extension when the check is finished.
  • In vCenter 7.0 Update 2 the version of vPostGres used by vCenter is upgraded from vPostGres 11 to vPostGres 12.
  • vPostGres 12 does not have the amcheck_next library, so the vPostGres upgrade fails if the amcheck_next extension is present.

Resolution

To resolve the issue, remove the amcheck_next extension from vCenter Database

  • The current version of the vPostgres Consistency Check script vpgsql-consistency-check.sh will remove the amcheck_next extension when it complete the check.

  • You can also remove the extension by connecting to the vCenter Database.
    1. Take a snapshot of the vCenter before making the changes. If the vCenter is part of an Enhanced Link Mode (ELM) configuration, the snapshots should be taken with the vCenters in a powered off state.
    2. SSH to the vCenter with root user.
    3. Connect to the vpostgres DB 
      /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
    4. Query the Database to confirm the amcheck_next extension is present. 
      select * from pg_extension;
    5. Remove the extension. 
      drop extension amcheck_next;
  • After removing the extension, retry the update/upgrade of vCenter.




Powered by Wolken Software