Failed vCenter Upgrade through SDDC Manager Due to Repository Configuration Error

Products

VMware SDDC Manager

Issue/Introduction

During an attempt to upgrade a VMware vCenter Server via SDDC Manager, users encountered errors related to repository configuration. The error message indicated that the update repository could not be set due to network issues or incorrect URL settings.

/var/log/vmware/vcf/lcm/lcm-debug.log on the SDDC manager:

Failed to set the repository https://<vcenter_fqdn>/vmware/vcf/bundle/96368be7-554c-40d1-81ef-3ad0b47c16f0/update_repo during upgrade for vCenter <vc_fqdn> due to exception NotFound (com.vmware.vapi.std.errors.not_found) => { messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => { id = com.vmware.appliance.update_error, defaultMessage = %s, args = [Network failure. Check the network settings and try again.], params = <null>, localized = <null> }], data = <null>, errorType = NOT_FOUND }

/var/log/vmware/applmgmt/PatchRunner.log on the vCenter:

INFO vpxd Copied vpxd cfg files to tmp
YY-MM-DDTHH:MM:SS. vpxd:Expand INFO vpxd Completed vpxd.cfg.tmp modification
YY-MM-DDTHH:MM:SS. vpxd:Expand ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpxd:Expand' failed.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatee43ps3cg/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/updatemgr/software-updatee43ps3cg/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)

Matches https://knowledge.broadcom.com/external/article?legacyId=96830, and the resolution in the KB does not resolve the issue.

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

After a failed vCenter upgrade attempt through SDDC Manager and re-trying the upgrade, vCenter VMDIR state goes into standalone mode and needs to be fixed.

Resolution

1. Verify and Correct Repository URL:
Ensure that the repository URL is correct and accessible. May need to manually set the URL in the vCenter Appliance Management Interface (VAMI - vcenterFQDN:5480 site) by doing the following:

Log in to the VAMI
Click the updates tab > Settings > Select specified URL > Change the URL to what SDDC is calling out
Retry the upgrade

2. Reset the VMDIR Service and clean up the failed upgrade files from vCenter by doing the following:

Take powered-off snapshots of all linked vCenters.
SSH into the vCenter as root and run the following command to modify the likewise registry to update the value depending on the vCenter Server versions:

For any version below 8.0 U3:
/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

For 8.0 U3 and above versions:
/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -L /var/log/vmware/vmdird/vmdird.log"

Then cleanup the upgrade files by running the following commands:

service-control --stop applmgmt

rm -rf /storage/core/software-update/updates/*
rm -rf /storage/core/software-update/stage/*
rm -rf /storage/db/patching.db
mv /etc/applmgmt/appliance/software_update_state.conf /storage/core

service-control --start applmgmt

Retry vCenter upgrade from SDDC Manager.