vCenter update stuck and failed at 80% with an Error "Exception occured in postinstallHook for vpostgres:Patch"
search cancel

vCenter update stuck and failed at 80% with an Error "Exception occured in postinstallHook for vpostgres:Patch"

book

Article ID: 431096

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • When attempting to patch or update the vCenter Server Appliance (vCSA) , the upgrade process fails and stuck at 80% with an Error: "Exception occurred in postinstallHook for vpostgres:Patch"
  • For example, the update fails when updating the vCenter server from 8.0.3 (24853646) to 8.0.3 (25092719)


  • The snippets below will be observed in the /var/log/vmware/applmgmt/PatchRunner.log file in vCenter server at the time of update failure indicates an Incorrect pgpass entry found for DB replicator user credentials:

    YYYY-MM-DDTHH:MM:SS.608Z  Running command: ['su', '-s', '/bin/bash', '-', 'vpostgres', '-c', '/opt/vmware/vpostgres/14/bin/pg_ctl reload -D /storage/db/vpostgres']
    YYYY-MM-DDTHH:MM:SS.811Z  Done running command
    YYYY-MM-DDTHH:MM:SS.812Z  Checking for password of user postgres, db postgres and port 5432 in /root/.pgpass
    YYYY-MM-DDTHH:MM:SS.812Z  Found password match for user postgres, db postgres, port 5432
    YYYY-MM-DDTHH:MM:SS.812Z  postgres password is 'password'
    YYYY-MM-DDTHH:MM:SS.812Z  Running: /opt/vmware/vpostgres/14/bin/psql -U postgres -c ALTER ROLE postgres WITH PASSWORD ''password'';
    YYYY-MM-DDTHH:MM:SS.812Z  Running command: ['/opt/vmware/vpostgres/14/bin/psql', '-U', 'postgres', '-c', "ALTER ROLE postgres WITH PASSWORD ''password'';"]
    YYYY-MM-DDTHH:MM:SS.834Z  Done running command
    YYYY-MM-DDTHH:MM:SS.834Z  Checking for password of user replicator, db replication and port 5432 in /root/.pgpass
    YYYY-MM-DDTHH:MM:SS.835Z  Removing temp file /tmp/file_name
    YYYY-MM-DDTHH:MM:SS.835Z  Removing temp file /tmp/file_name
    YYYY-MM-DDTHH:MM:SS.835Z  Removing temp file /tmp/file_name
    YYYY-MM-DDTHH:MM:SS.835Z  Removing temp file /tmp/file_name
    YYYY-MM-DDTHH:MM:SS.835Z  Removing temp file /tmp/file_name
    YYYY-MM-DDTHH:MM:SS.836Z  Removing temp file /tmp/file_name
    , stderr: YYYY-MM-DDTHH:29:06.867Z  Incorrect pgpass entry found:
    YYYY-MM-DDTHH:MM:SS.835Z  Exception: Traceback (most recent call last):
      File "/opt/vmware/vpostgres/current/bin/vmw_vpg_config/vmw_vpg_config.py", line 314, in main
        return act(opts)
      File "/opt/vmware/vpostgres/current/bin/vmw_vpg_config/vmw_vpg_config.py", line 308, in act
        return UpgradeInstanceInplace(cfg)
      File "/opt/vmware/vpostgres/14/share/python-modules/vpostgres_cis/vcva.py", line 3525, in UpgradeInstanceInplace
        ret = UpgradeInPlacePost(cfg, upgrade_kind=ret)
      File "/opt/vmware/vpostgres/14/share/python-modules/vpostgres_cis/vcva.py", line 3271, in UpgradeInPlacePost
        ret = UpdateAuthPostUpgrade(cfg)
      File "/opt/vmware/vpostgres/14/share/python-modules/vpostgres_cis/vcva.py", line 3571, in UpdateAuthPostUpgrade
        repl_pw = GetPgPassConfPasswd(cfg['PGPASS_CONF'], 'replication',
      File "/opt/vmware/vpostgres/14/share/python-modules/vpostgres_cis/vcva.py", line 635, in GetPgPassConfPasswd
        if entry[1] == port and \
    IndexError: list index out of range
    ..
    YYYY-MM-DDTHH:MM:SS.859Z vpostgres:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpostgres:Patch' failed.
    Traceback (most recent call last):
    ..
    patch_errors.UserError: PostgreSQL failed to run upgrade command.
    YYYY-MM-DDTHH:MM:SS.863Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
    ..
    YYYY-MM-DDTHH:MM:SS.867Z WARNING root stopping status aggregation...
    YYYY-MM-DDTHH:MM:SS.868Z ERROR __main__ Patch vCSA failed

  • In some cases the /var/log/vmware/applmgmt/PatchRunner.log file in vCenter server at the time of update failure indicates /root/.pgpass is missing.

    YYYY-MM-DDTHH:MM:SS.408Z  Done running command
    YYYY-MM-DDTHH:MM:SS.408Z  Done to change path /var/run/vpostgres owner to vpostgres
    YYYY-MM-DDTHH:MM:SS.408Z  Running command: ['chown', '-R', 'vpostgres:vpgmongrp', '/storage/db/vpostgres_ssl']
    YYYY-MM-DDTHH:MM:SS.409Z  Done running command
    YYYY-MM-DDTHH:MM:SS.409Z  Done to change path /storage/db/vpostgres_ssl owner to vpostgres
    YYYY-MM-DDTHH:MM:SS.410Z  Running command: ['chown', '-R', 'vpostgres:vpgmongrp', '/dev/shm/postgres_stats']
    YYYY-MM-DDTHH:MM:SS.411Z  Done running command
    YYYY-MM-DDTHH:MM:SS.411Z  Done to change path /dev/shm/postgres_stats owner to vpostgres
    YYYY-MM-DDTHH:MM:SS.418Z  Updated /storage/db/vpostgres/postgresql.conf
    YYYY-MM-DDTHH:MM:SS.419Z  Removing temp file /tmp/tmpfuv1udj1
    YYYY-MM-DDTHH:MM:SS.419Z  Removing temp file /tmp/tmp1s23b0yo
    YYYY-MM-DDTHH:MM:SS.419Z  Removing temp file /tmp/tmp5hhaj2de
    YYYY-MM-DDTHH:MM:SS.419Z  Removing temp file /tmp/tmpv7k6pgnk
    YYYY-MM-DDTHH:MM:SS.419Z  Removing temp file /tmp/tmpq9ypk66e
    , stderr: YYYY-MM-DDTHH:MM:SS.418Z  Expected file /root/.pgpass is missing
    YYYY-MM-DDTHH:MM:SS.419Z  could not find entry in /root/.pgpass for user postgres, db VCDB
    .
    YYYY-MM-DDTHH:MM:SS.459Z vpostgres:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpostgres:Patch' failed.
    Traceback (most recent call last):
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
        executionResult = systemExtension(args)
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
        result = self.extension(*args)
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
        return func(*args)
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/payload/components-script/vpostgres/__init__.py", line 256, in patch
        raise UserError(_(_T('vpostgres.patch.upgrade',patch_errors.UserError: PostgreSQL failed to run upgrade command.
    YYYY-MM-DDTHH:MM:SS.468Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
    Traceback (most recent call last):
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
        _patchComponents(ctx, userData, statusAggregator.reportingQueue)
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents
        executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook
        result = executeHook(c.patchScript, hook, args,
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
        result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
      File "/storage/seat/software-update2ad1_aty/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
        raise expatch_errors.ComponentError
    YYYY-MM-DDTHH:MM:SS.473Z WARNING root stopping status aggregation...
    YYYY-MM-DDTHH:MM:SS.475Z ERROR __main__ Patch vCSA failed

Environment

vCenter 8.x

Cause

This issue occurs because the /root/.pgpass configuration file is missing the required authentication entries for the internal database user named 'replicator'.
During the upgrade, the vcva.py script attempts to parse this file to validate the credentials for both the postgres and replicator users.
When it fails to find the replicator entries, the script encounters an unhandled IndexError and crashes, halting the entire upgrade process.

Resolution

Note: Before proceeding with the Resolution steps below, take a valid snapshot of the vCenter Server Appliance. If the vCenter server is a part of ELM- Enhanced Linked Mode, take offline snapshots of each vCenter server In the ELM).

Step 1: Temporarily Bypass Database Authentication

     SSH into the vCenter Server Appliance as root.

     Backup the PostgreSQL configuration file:
     # cp /storage/db/vpostgres/pg_hba.conf /storage/db/vpostgres/pg_hba.conf.bak

     Edit the file:
   # vi /storage/db/vpostgres/pg_hba.conf

     Find the local connection lines and change the authentication method from md5 to trust:

     Change "host all all 127.0.0.1/32 md5" to "host all all 127.0.0.1/32 trust"

     Change "local all all md5" to "local all all trust"

     Save and close the file (:wq!).

     Restart the vPostgres service to apply the changes:
   # service-control --restart vpostgres

Step 2: Reset Database User Passwords ( for both postgres and replicator users )

     Connect to the database (it will bypass the password prompt due to the trust setting):
   # /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

     Reset the passwords for both the postgres and replicator users. Ensure you enclose the new password in single quotes:
     # alter user postgres with password 'NEW_PASSWORD1';
     # alter user replicator with password 'NEW_PASSWORD2';

     Exit the database prompt:
   # \q

Step 3: Edit the .pgpass File.

      Recreate the .pgpass file and set the correct strict permissions:

      # touch /root/.pgpass
   # chmod 0600 /root/.pgpass

     Edit .pgpass file ( # vi /root/.pgpass) and remove/delete all existing entries and insert below mentioned 8 lines.

   /var/run/vpostgres:5432:*:postgres:NEW_PASSWORD1
   localhost:5432:postgres:postgres:NEW_PASSWORD1
   127.0.0.1:5432:postgres:postgres:NEW_PASSWORD1
   localhost:5432:VCDB:postgres:NEW_PASSWORD1
   127.0.0.1:5432:VCDB:postgres:NEW_PASSWORD1
   localhost:5432:replication:replicator:NEW_PASSWORD2
   127.0.0.1:5432:replication:replicator:NEW_PASSWORD2
   /var/run/vpostgres:5432:replication:replicator:NEW_PASSWORD2

     Save the file (:wq!)

Step 4: Reinstate Database Authentication.

   Revert the pg_hba.conf file from Step 1 to revert trust to the original md5 value:
      # cp /storage/db/vpostgres/pg_hba.conf.bak /storage/db/vpostgres/pg_hba.conf

   Restart the vPostgres service to apply the changes:
   # service-control --restart vpostgres

Step 5: Resume Upgrade.

Additional Information

Normally the password for role 'postgres' and 'replicator' are generated with random password of length 16 to meet below policy:

  1. Minimal length is no less than 8 and max length limit to 16.
  2. Make sure there is at least one character from each character sets (lower case, upper case, special character and numbers) .
  3. Ensure password does not have repeated or consecutive identical characters.
Powered by Wolken Software