vCenter upgrade fails at vCenter upgrade set repo stage - VMware Cloud Foundation

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
vCenter upgrade fails at vCenter upgrade set repo stage 

Environment

VMware Cloud Foundation 4.x

Cause

This issue is caused due to corruption in RPM Database on vCenter Server Appliance.

lcm logs indicate following :

ERROR [vcf_lcm, 9e43dde85f 1137d, 8 f 14, upgradeId=c851, resourceType=le9876202179, [c.v.e.s.l.p.] to set the repository vmware/re/vcf/bundle/fab032e8c696/update_repo -

Exception: InternalServerError
Cause: [vmware.vapi.std.errors]
Details:
— LocalizableMessage {
   - messages.id = vapi.method.invoke.exception
   - defaultMessage = Error in method invocation
   - rpmdb open failed
}
Params:
- localized.data = errorType = INTERNAL SERVER ERROR
- REOR-VCENTER: during upgrade for vCenter
- Error Message: VCenter REPO FAILED 
- Timestamp: 53:20.569+0000

INFO: [vcf_lcm, le9876202179, bundleElementId=72698882-175d-4954-a57c-S97g7b6d21b5] 
INFO: [vcf_lcm, 9e4 ö3dde35f 1137d, 3 f 14, upgradeId=c851, resourceType=VCENTER, resourceId=ab5bfdb4-4e7c-a3d1] 
- VCenter upgrade failed with VCENTER UPGRADE SET REPO 9e4 6Bdde8Sf1197d
- UpgradeId=c851 Se, resourceType=VCENTER, resourceId=ab5bfB84-814b-4S7c-a3dE
- VC VIM connection for: 63cb3cace11f4630, f848

Resolution

1. Take a snapshot of the vCenter Server Appliance and  SDDC VM.
2. Take SSH/Putty session to the vCenter Server Appliance
3. Run the command:
   • rm -f /var/lib/rpm/__db*
   • rpm -qa  [To Rebuild RPM database run this query]
4. Restart vCenter VM
5. Re-run  vCenter upgrade from  SDDC Manager UI