rpm database corruption causes Asyn Patch Tool patch enablement operation to fail

Products

• VMware Cloud Foundation

Issue/Introduction

This KB is designed to assist in recovering from the rpm database corruption and allowing the customer to proceed with patch enablement.

Symptoms:

• Enabling a patch using the Async Patch Tool fails:

 

vcf-async-patch-tool -e --patch VCENTER:0.0.0.0-12345678 --sddcSSOUser [email protected] --sddcSSHUser vcf --depotUser [email protected] --outputDirectory /nfs/vmware/vcf/nfs-mount/apToolBundles/ --it ONLINE

 

• The tool displays errors similar to below:

 

Error Message: VCF service upgrade or config property update script failed with error Running the following command(s) as su user has failed : python /home/vcf/asyncPatchTool/conf/vcfserviceupgrade/vcf-multi-service-rpm-upgrade.py --inputData '[{\"name\":\"lcm\",\"package_name

 

Remedy: Please ensure the config update files (if present) are accurate. Refer to the upgrade_status.json in your current directory for the status of service upgrades and the script logs located at /home/vcf/asyncPatchTool/bin/async_patch_tool.log

 

• The /home/vcf/asyncPatchTool/bin/async_patch_tool.log contains entries similar to below:

 

Starting SDDC Manager service updates. This process might take a few minutes.

2024-04-05 13:15:41.315 [main] DEBUG [com.vmware.vcf.shared.annotations.validators.FQDNValidator]

         FQDN validator failed match for value: CENSORED

................................

2024-04-05 13:15:46,742 : Execute cmd: rpm -Uvh --replacepkgs /nfs/vmware/vcf/nfs-mount/apToolBundles/sddc-bundle-files/bundle-74965/sddc-manager-upgrade/vcf

-lcm-4.4.1-21506568.pak 2>&1

2024-04-05 13:15:46,755 : error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 - (-30973)

error: cannot open Packages database in /var/lib/rpm

error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 - (-30973)

error: cannot open Packages database in /var/lib/rpm

 

• When running the command rpm -qa you do not see the expected output but rather you see more database related errors:

 

Environment

VMware Cloud Foundation 4.x

Resolution

Workaround:

Snapshot the SDDC Manager - DO NOT SKIP THIS STEP

 

Backup the contents of /var/lib/rpm:

 

cp -a /var/lib/rpm /home/vcf/rpm/

 

Delete existing files:

 

rm /var/lib/rpm/__db.*

 

Rebuild the RPM DB:

 

rpm --rebuilddb

 

List the RPMs again and make sure expected output is present

 

rpm -qa

 

Attempt the patch enablement once more.

Additional Information

Impact/Risks:

HIGH - the fix involves manipulation of the rpm database. A snapshot of the SDDC Manager is MANDATORY before proceeding.