vCenter Async Patch 7.0U3o fails with error, "Installation of one or more RPMs failed"

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

vCenter Async Patch 7.0U3o patch fails with the following error in SDDC, "Software packages failed to install"
This issue impacts vCenters that have undergone an upgrade from VC 7.0 to VC 7.0U3 (any patch version earlier than U3i) and have been subsequently updated to 7.0U3o. We recommend following the pre-update remediation steps in this KB before applying the 7.0U3o patch. 

You can check the upgrade sequence by running the following command from the appliance shell on the vCenter. To change shell from bin/bash run command appliancesh

software-packages list --history

Example of an affected upgrade sequence:
Command> software-packages list --history
[2023-11-15 20:17:39,387] :
'Name'                                  'Install Date'
VC701MP1-HP-2755530         2021-07-02T16:45:13.576Z
VC-7.0U2d                               2021-10-13T06:22:19.013Z
VC-7.0U3d                               2022-10-17T20:59:48.758Z

Similar errors are seen in the lcm.log on the SDDC Manager. 

/var/log/vmware/vcf/lcm/lcm.log

2023-11-09T18:22:38.219+0000 ERROR [vcf_lcm,94e2e03a9d213e29,d7d0,upgradeId=8878fa31-f923-43a7-94e0-17484a11a93e,resourceType=
VCENTER,resourceId=6caa0bc9-20dd-4dd3-985c-b0d7dccf6f36,bundleElementId=a72c790f-66e1-430e-aa0c-682692ffd3ac] [c.v.e.s.l.p.i.v
center.VCenterCommon,Async-4]
VCENTER FQDN software packages failed to install. Update info Info (com.vmware.appliance.update.info) => {
    state = UP_TO_DATE,
    task = <null>,
    version = 7.0.3.00500,
    latestQueryTime = 10/12/22 06:56:02 +0000
}

Similar errors are seen in the update_microservice.log on the vCenter.

/var/log/vmware/applmgmt/update_microservice.log
Updating / installing...
bash-4.4.18-4.ph3 ########################################
Cleaning up / removing...
bash-4.4.18-2.ph3 ########################################
][/bin/sh: error while loading shared libraries: libtinfo.so.6: cannot open shared object file: No such file or directory
warning: %postun(bash-4.4.18-2.ph3.x86_64) scriptlet failed, exit status 127

Environment

VMware Cloud Foundation 4.x

Cause

The problem stems from a bash-rpm update in 7.0U3o, causing a regression in the vCenter 7.0U3o branch.

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:

Failed Update Recovery Steps:

 1. Revert the failed vCenter to a previously working version
 2. SSH into the vCenter server using root credentials.
 3. Remove the stale Patching.db file from vCenter
    mv /storage/db/patching.db /storage/db/patching.db.old
 4. Retry the update to 7.0U3o from the SDDC UI. 
 

Pre-Update Remediation:

1.SSH into the vCenter(s)
2.Check for the patching.db*
ls -lh /storage/db/patching.db
3.Remove the stale Patching.db file from the vCenter(s) 
mv /storage/db/patching.db /storage/db/patching.db.old
4.Verify the file is no longer present:
ls -lh /storage/db/patching.db
5.Start the upgrade from the SDDC UI.