Aria Operations for Logs (Formerly vRealize Log Insight) & vRealize Suite Lifecyce Manager upgrade fails due to rpm database corruption

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:
You may experience one or more of the below symptoms.

• The Aria Operations for Logs (Formerly vRealize Log Insight) upgrade fails with the error:

Failed to upgrade: Failed to read installed version: error: rpmdb: BDB0113 Thread/process 16462/140029738584256 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm error: rpmdb: BDB0113 Thread/process 16462/140029738584256 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm

• Uploading the Aria Operations for Logs (Formerly vRealize Log Insight) upgrade PAK hangs.  No error is given, and the license accept page does not appear.  The /storage/core/loginsight/var/upgrade.log stops logging after loginsight-upgrade INFO Signature of the manifest validated: Verified OK.
• For vRealize Suite Lifecyce Manager, we see these messages in /opt/vmware/var/log/vami/updatecli.log
  
  13/03/2025 13:39:55 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/2/test_command
error: rpmdb: BDB0113 Thread/process 4907/139852798203712 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/Linux-PAM-1.3.0-1.ph3.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 66fd4949: NOKEY
error: rpmdb: BDB0113 Thread/process 4907/139852798203712 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
13/03/2025 13:39:55 [ERROR] Failed with exit code 256
  
  

Environment

Aria Operations for Logs 8.x

vRealize Log Insight 8.x

vRealize Suite Lifecyce Manager 8.x

Cause

This issue is cause by corruption in the rpm database.

Resolution

To resolve the issue, rebuild the rpm database.

1. Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Verify one or more rpm database files are present on the node by running the following command:
   

   ls -ltha /var/lib/rpm

   
   Note: If no _db.* files are listed in the directory, the node can be skipped and no further actions are required. Proceed to the next node.
   
   
3. Run the following command to create a backup directory for the rpm:
   

   mkdir /var/lib/rpm/backup

4. Run the following command to backup the rpm data:
   

   cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/

5. Delete the old database files:
   

   rm -f /var/lib/rpm/__db.[0-9][0-9]*

6. Run the following commands to list all rpms:
   

   rpm --quiet -qa

   Note: If this command hangs, cancel out of it with CTRL+C and continue on to the next step.

6. Run the following command to rebuild the rpms:
   

   rpm --rebuilddb

   Note: If this command returns Error: could not delete old database, this is benign and you may proceed to the next step

7. Run the following command to remove all cached data:
   

   yum clean all

8. Repeat steps 1-7 on all other nodes in the cluster. 

Now that the rpm database has been rebuilt, attempt the Aria Operations for Logs (Formerly vRealize Log Insight) upgrade again.