Aria Operations Upgrade Fails at Stage 3 Due to Postgres Replication Directory Error

Products

• VMware Aria Suite

Issue/Introduction

During the upgrade of Aria Operations, the process is getting stuck at stage 3 of 14. The issue arises specifically during the database upgrade phase, where the vPostgresReplication setup fails with a "directory not empty" error. This halts the upgrade process, preventing successful completion.

The DB upgrade has failed on the node:

/var/log/vmware/vcops/vcops_upgrade_XXXXXX_XXX.log

Traceback (most recent call last):
  File "/usr/lib/vmware-vcops/user/conf/install/vPostgresReplDatabaseSetup.py", line 52, in install
    exit_code = VPostgresReplDatabaseSetup.__linux_setup_vpostgres_repl()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/vmware-vcops/user/conf/install/vPostgresReplDatabaseSetup.py", line 40, in __linux_setup_vpostgres_repl
    VPostgresReplDatabaseSetup.__execute_process(vPostgresDbSetupFileName)
  File "/usr/lib/vmware-vcops/user/conf/install/vPostgresReplDatabaseSetup.py", line 34, in __execute_process
    raise RuntimeError('Error occurred calling {0} {1}'.format(cmd, script_results))
RuntimeError: Error occurred calling /usr/lib/vmware-vcops/persistence/vpostgres_repl_db_setup.sh SubprocessResponse(success=False, rc=1, stderr='', stdout="Fri Oct 25 15:32:29 IST 2024\nprocess id: 10717\nvRealize Operations vPostgres Replication directory /storage/db/vcops/vpostgres/repl\n/storage/db/vcops/vpostgres/repl is not empty. Use 'force' argument to wipe data directory.\n")
2024-10-25T10:02:29 ERROR - VPostgresReplDatabaseSetup.install() failed with rc=1

Environment

Aria Operations 8.x 

Cause

This error occurs because the vPostgresReplication database setup script (vPostgresReplDatabaseSetup.py) checks if the replication directory /storage/db/vcops/vpostgres/repl is empty. If any file or even a single log file is present, the script will abort the upgrade process to avoid potential data loss. 

Resolution

To resolve the issue, perform the following steps on the affected node:

Revert the Cluster to Pre-Upgrade State using vsphere snapshots. Ensure the cluster is reverted to the state it was in before the upgrade attempt.

Take a new snapshot before performing the below operation:

    1. Remove Existing Replication Directory. Open a shell session to the affected node and run the following command to remove the non-empty replication directory:

              rm -rf /storage/db/vcops/vpostgres/repl

    2. After the directory has been removed, execute the following Python script to reinitialize the vPostgres replication setup:

             python /usr/lib/vmware-vcops/install/vPostgresReplDatabaseSetup.py

       Note: There is no need to restart any services after this step.

Once the setup script has completed successfully, restart the upgrade process. The upgrade should proceed beyond stage 3 without encountering the directory error.

 

Additional Information

The vPostgres replication directory contains critical database replication files used for Aria Operations' central database. The setup script will not proceed if the directory is not empty to prevent accidental overwriting of important data. This procedure helps ensure that no important data is lost during the upgrade.