Aria Operations for Logs (Formerly vRealize Log Insight) upgrade fails due to rpm database corruption

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
You may experience one or more of the below symptoms.

• The Aria Operations for Logs (Formerly vRealize Log Insight) upgrade fails with the error:

Failed to upgrade: Failed to read installed version: error: rpmdb: BDB0113 Thread/process 16462/140029738584256 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm error: rpmdb: BDB0113 Thread/process 16462/140029738584256 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm

• Uploading the Aria Operations for Logs (Formerly vRealize Log Insight) upgrade PAK hangs.  No error is given, and the license accept page does not appear.  The /storage/core/loginsight/var/upgrade.log stops logging after loginsight-upgrade INFO Signature of the manifest validated: Verified OK.

Environment

Aria Operations for Logs 8.x

vRealize Log Insight 8.x

Cause

This issue is cause by corruption in the rpm database.

Resolution

To resolve the issue, rebuild the rpm database.

1. Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Verify one or more rpm database files are present on the node by running the following command:
   

   ls -ltha /var/lib/rpm

   
   Note: If no _db.* files are listed in the directory, the node can be skipped and no further actions are required. Proceed to the next node.
   
   
3. Run the following command to create a backup directory for the rpm:
   

   mkdir /var/lib/rpm/backup

4. Run the following command to backup the rpm data:
   

   cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/

5. Delete the old database files:
   

   rm -f /var/lib/rpm/__db.[0-9][0-9]*

6. Run the following commands to list all rpms:
   

   rpm --quiet -qa

   Note: If this command hangs, cancel out of it with CTRL+C and continue on to the next step.

6. Run the following command to rebuild the rpms:
   

   rpm --rebuilddb

   Note: If this command returns Error: could not delete old database, this is benign and you may proceed to the next step

7. Run the following command to remove all cached data:
   

   yum clean all

8. Repeat steps 1-7 on all other nodes in the cluster. 

Now that the rpm database has been rebuilt, attempt the Aria Operations for Logs (Formerly vRealize Log Insight) upgrade again.