vCenter 8.0.3 fails to update with error: "Installation failed. Please collect the VC support bundle."

Products

VMware vCenter Server

Issue/Introduction

vCenter patch update install fails from vCenter CLI with error message.

[YYYY-MM-DD HH:MM:SS] : Installing version: 8.0.3.#####
[YYYY-MM-DD HH:MM:SS] : Installation failed. Please collect the VC support bundle.
vCenter update via VAMI fails with the following error message:

Installation of one or more RPMs failed.
You have reached maximum number of retries to resume the patching. Please restore the vCenter using the backup.
The installation cannot be resumed, revert to the latest vCenter backup
patchRunner.log (Path: /var/log/vmware/applmgmt/) reported VMDir replication not working correctly.

"name": "vmdir",
"patchScript": "/storage/seat/software-updatedvsay6u1/stage/scripts/patches/payload/components-script/vmdir",
"requirementsResult": {
"mismatches": [
{
"description": {
"id": "vmdir.replicationerror.text",
"localized": "VMDir replication is not working correctly",
"translatable": "VMDir replication is not working correctly"
},
"problemId": null,
"relatedUserDataId": null,
"resolution": {
"id": "vmdir.replicationerror.resolution",
"localized": "Please refer the KB to handle VMDir replication issues: https://kb.vmware.com/s/article/93910",
"translatable": "Please refer the KB to handle VMDir replication issues: https://kb.vmware.com/s/article/93910"
},
"severity": "error",
"text": {
"id": "vmdir.replicationerror.text",
"localized": "VMDir replication is not working correctly",
"translatable": "VMDir replication is not working correctly"
}
}
update_microservice.log (Path: /var/log/vmware/applmgmt/) reports VMDir Replication is not working between partners.

"text": {
"id": "vmdir.collect_requirements.vmdirstalevcs.text",
"translatable": "VMDir Replication is not working between partners as they are not reachable. If they are stale please decommission them.",
"localized": "VMDir Replication is not working between partners as they are not reachable. If they are stale please decommission them."
},
"description": {
"id": "vmdir.collect_requirements.vmdir.description",
"translatable": "",
"localized": ""
},
"resolution": {
"id": "vmdir.collect_requirements.vmdirstalevcs.resolution",
"translatable": "Non-reachable vCenter servers are %(0)s. Make sure the stale vCenter servers are decommissioned and all vCenter servers are reachable before continuing. Refer KB %(1)s to decommission the stale vCenter servers.",
"args": [
"########",
"https://knowledge.broadcom.com/external/article/316470"
],
"localized": "Non-reachable vCenter servers are ########. Make sure the stale vCenter servers are decommissioned and all vCenter servers are reachable before continuing. Refer KB https://knowledge.broadcom.com/external/article/316470 to decommission the stale vCenter servers."
},
"problemId": null,
"severity": "warning",
"relatedUserDataId": null
vCenter could be previously part of an ELM (Enhanced linked mode) and later made into a standalone vCenter.
Significant slowness during vSphere Client login can be observed while stale vCenter is present in the environment.

Environment

vCenter 8.x

Cause

Stale SSO sites, replication agreements, or expired certificates from decommissioned vCenter nodes remain in the VMDir database. The update process requires all known partners to be reachable before proceeding.

Resolution

A. Validate if stale vCenter entries are still present within vmdir:

SSH to the existing vCenter via root
Run the following command from all available vCenter servers to view the replication partner status:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u Administrator

Example:
root@VC1 [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u Administrator
password:
Partner: VC2.example.com
Host available: Yes
Status available: Yes
My last change number: 1752497
Partner has seen my change number: 1383370
Partner is 369127 changes behind.
root@VC2 [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u Administrator
password:
Partner: VC1.example.com
Host available: Yes
Status available: Yes
My last change number: 1556148
Partner has seen my change number: 1479071
Partner is 77077 changes behind.

B. If there is still a replication agreement between the current vCenter and an old vCenter that no longer exists, this indicates the old vCenter was not properly decommissioned.

Note: Prior to making any changes on the vCenter Server, take a snapshot of the vCenter Appliance. If the vCenter Server is in linked mode, take offline snapshots of all the vCenter Servers in the linked mode, refer: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Unregister the stale vCenter:

cmsso-util unregister --node-pnid <old_vCenterServer_System_Name> --username [email protected] --passwd 'vCenter_Single_Sign_On_password'

where old_vCenterServer_System_Name is the FQDN or IP address of the vCenter Server that is to be decommissioned. Enter the vCenter_Single_Sign_On_password in quotes. Reference KB Using the cmsso command to unregister vCenter with External PSC or vCenter with Embedded PSC from Single Sign-On.

C. If the cmsso-utils command does not remove the entries completely, follow the below workaround to manually delete the entries and unstage the vCenter update:

Take powered-off snapshots of all the vCenter servers in the ELM or the standalone vCenter, and remove the replication agreements using the below command and reboot the vCenter.
Referencing Determining replication agreements and status with the Platform Services Controller (PSC), run the below command to remove the replication agreement:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f removeagreement -2 -h <Source_PSC/vCenter_FQDN> -H <PSC/vCenter_FQDN_to_Remove_from_Replication> -u administrator -w 'Administrator_Password'
Manually un-stage the update, re-stage the update from VAMI, and install the update.

1. SSH to the vCenter via root
2. Unstage the update:
  
  software-packages unstage
3. Start the update process again.

D. Use this alternative method to clean up the staged vCenter update if unstage command fails.

Take powered-off snapshots of all the vCenter either in the ELM or standalone. Reference KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.
SSH into vCenter via root
Stop the appliance management service:
service-control --stop applmgmt
Clean up the different directories:
rm -rf /storage/core/software-update/updates/*rm -rf /storage/core/software-update/stage/*rm -rf /storage/db/patching.dbmv /storage/core/software-packages/staged-configuration.json /storage/coremv /etc/applmgmt/appliance/software_update_state.conf /storage/core
Restart the appliance management service:

service-control --start applmgmt
If the system uses the /storage/seat partition to store the software-update, clean it up with the command:
rm -rf /storage/seat/software-*
Start the update process.

If further assistance is required, contact Broadcom Technical Support.

Additional Information

Also refer: Update of vCenter fails with "Stage path file doesn't exist" or "Package discrepency error, Cannot resume!"

Use the lsdoctor tool to perform a comprehensive health check of the lookup service.