"Error in method invocation rpmdb open failed", error while upgrading vCenter Server Appliance 6.x/7.x

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrading vCenter Server Appliance fails with following error messages :

"Error in method invocation rpmdb open failed"
or
"Cannot open Packages database in /var/lib/rpm"

• In a VMware Cloud Foundation (VCF) environment, SDDC Manager will show following error message for the vCenter Server patching failure

Failed to set the repository https://<fqdn>/vmware/vcf/bundle/<id>/<id>/update_repo during upgrade for vCenter <fqdn> due to exception 500 Internal Server Error: """"{""""type"""":""""com.vmware.vapi.std.errors.internal_server_error"""",""""value"""":{""""error_type"""":""""INTERNAL_SERVER_ERROR"""",""""messages"""":[{""""args"""":[""""rpmdb open failed""""],""""default_message"""":""""Error in method invocation rpmdb open failed"""",""""id"""":""""vapi.method.invoke.exception""""}]}}""""
 

• Patch update from command line might show following error message : 

error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm

• Appliance Management log (/var/log/vmware/applmgmt.log) will show error messages similar to below entries :

2022-06-11T20:57:36.408 [3038]DEBUG:vmware.appliance.update.update_b2b:Listing completed successfully
2022-06-11T20:57:36.408 [3038]DEBUG:vmware.appliance.update.update_b2b:List returns []
2022-06-11T20:57:36.408 [3038]DEBUG:vmware.appliance.update.update_pending:Got exception during listing updates rpmdb open failed
2022-06-11T20:57:36.408 [3038]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.appliance.update.pending in list - rpmdb open failed
Traceback (most recent call last):
 File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
  service_id, operation_id, input_value, ctx)
 File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 248, in _invoke_int
 File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1526, in _discoverUpdateAt
  installedPkgs = _getInstalledPackages()
 File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 386, in _getInstalledPackages
  matchIter = transactionSet.dbMatch()
_rpm.error: rpmdb open failed

 

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue is caused due to corruption in RPM Database on vCenter Server Appliance.

Resolution

Please follow below steps to resolve this issue :

1. Take a snapshot of the vCenter Server Appliance.
2. Take SSH/Putty session to the vCenter Server Appliance
3. Run the command:
   • rm -f /var/lib/rpm/__db*
   • rpm -qa  [To Rebuild RPM database run this query]

Sample Screenshot:

4. Restart the vCenter Server Appliance.