vCenter upgrade fails at vCenter upgrade set repo stage - VMware Cloud Foundation

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• vCenter upgrade fails at vCenter upgrade set repo stage 

Environment

• VMware Cloud Foundation 4.x

Cause

• This issue is caused due to corruption in RPM Database on vCenter Server Appliance.
• /var/log/vmware/vcf/lcm/lcm.log indicate following :
  
  

ERROR [vcf_lcm, 9e43dde85f 1137d, 8 f 14, upgradeId=c851, resourceType=le9876202179, [c.v.e.s.l.p.] to set the repository vmware/re/vcf/bundle/fab032e8c696/update_repo -

Exception: InternalServerError
Cause: [vmware.vapi.std.errors]
Details:
— LocalizableMessage {
   - messages.id = vapi.method.invoke.exception
   - defaultMessage = Error in method invocation
   - rpmdb open failed
}
Params:
- localized.data = errorType = INTERNAL SERVER ERROR
- REOR-VCENTER: during upgrade for vCenter
- Error Message: VCenter REPO FAILED 
- Timestamp: 53:20.569+0000

INFO: [vcf_lcm, le9876202179, bundleElementId=72698882-175d-4954-a57c-S97g7b6d21b5] 
INFO: [vcf_lcm, 9e4 ö3dde35f 1137d, 3 f 14, upgradeId=c851, resourceType=VCENTER, resourceId=ab5bfdb4-4e7c-a3d1] 
- VCenter upgrade failed with VCENTER UPGRADE SET REPO 9e4 6Bdde8Sf1197d
- UpgradeId=c851 Se, resourceType=VCENTER, resourceId=ab5bfB84-814b-4S7c-a3dE
- VC VIM connection for: 63cb3cace11f4630, f848

Resolution

1. Take a snapshot of the vCenter Server Appliance and  SDDC VM.
2. Take SSH/Putty session to the vCenter Server Appliance
3. Run the command:

   • rm -f /var/lib/rpm/__db*

   • rpm -qa       #To Rebuild RPM database run this query

4. Restart vCenter VM
5. Re-run  vCenter upgrade from  SDDC Manager UI