vCenter patch update initiated from SDDC Manager fails at the stage 'VCENTER_UPGRADE

Products

VMware SDDC Manager

Issue/Introduction

Checked the upgrade status in SDDC UI, failing with an error,
'vcenter upgrade invocation failed'

Below are the log entries we see in /var/log/vmware/vcf/lcm/lcm.log

YYYY-MM-DDTHH:MM:SS.727+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=f19c00f2-6bae-41d4-90e1-ec5692be2a42,resourceType=VCENTER,resourceId=a6a80d2e-171f-4f50-bc94-f34dab537697,bundleElementId=3eab1127-07de-4acc-a2fa-df3163f8b
e1a] [c.v.v.vapi.vsphere.VcenterVapiHelper,Upgrade-6] Exception occurred during VC vAPI invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.NotFound: NotFound (com.vmware.vapi.std.errors.not_found) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.appliance.update_error,
defaultMessage = %s,
args = [Network failure. Check the network settings and try again.],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = NOT_FOUND
}
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)

YYYY-MM-DDTHH:MM:SS.731+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=f19c00f2-6bae-41d4-90e1-ec5692be2a42,resourceType=VCENTER,resourceId=a6a80d2e-171f-4f50-bc94-f34dab537697,bundleElementId=3eab1127-07de-4acc-a2fa-df3163f8be1a] [c.v.e.s.l.p.i.v.VCenterRDUUpgrader,Upgrade-6] Invocation of RDUClient upgrade method failed due to exception :
com.vmware.evo.sddc.lcm.model.error.LcmException: null
..
..
YYYY-MM-DDTHH:MM:SS.748+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=f19c00f2-6bae-41d4-90e1-ec5692be2a42,resourceType=VCENTER,resourceId=a6a80d2e-171f-4f50-bc94-f34dab537697,bundleElementId=3eab1127-07de-4acc-a2fa-df3163f8b
e1a] [c.v.e.s.l.p.i.v.VCenterRDUPrimitiveImpl,Upgrade-6] Exception occurred during RDU execution at stage VCENTER_UPGRADE_PRECHECK
YYYY-MM-DDTHH:MM:SS.748+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=f19c00f2-6bae-41d4-90e1-ec5692be2a42,resourceType=VCENTER,resourceId=a6a80d2e-171f-4f50-bc94-f34dab537697,bundleElementId=3eab1127-07de-4acc-a2fa-df3163f8b
e1a] [c.v.e.s.l.p.i.v.VCenterRDUPrimitiveImpl,Upgrade-6] Exception occurred while performing upgrade
com.vmware.evo.sddc.lcm.model.error.LcmException: null

Environment

VCF 5.x

SDDC 5.x

Cause

When checked in /var/log/vmware/applmgmt/applmgmt.log in vCenter server.

applmgmt.log
YYYY-MM-DDTHH:MM:SS PM UTC [47701]DEBUG:vmware.appliance.update.update_functions:Running /usr/bin/wget --server-response --tries 3 --waitretry 1 --connect-timeout 10 -r -np -nH -nd -A -P /storage/core/software-update/tmp/latest https://sddcfqdn/vmware/vcf/bundle/VMware/vRack/lcm/8745914e-695d-41b3-95f1-8fd049522036/a6a80d2e-171f-4f50-bc94-f34dab537697/iso/repos/patcher_repo/manifest/manifest-latest.xml -e CENSORED -e CENSORED
YYYY-MM-DDTHH:MM:SS PM UTC [47701]DEBUG:vmware.appliance.update.update_functions:runCommandAndCheckResult failed: '--2025-03-14 18:25:47-- https://sddcfqdn/vmware/vcf/bundle/VMware/vRack/lcm/8745914e-695d-41b3-95f1-8fd049522036/a6a80d2e-171f-4f50-bc94-f34dab537697/iso/repos/patcher_repo/manifest/manifest-latest.xml\nConnecting to proxy_server_IP:8080... connected.\nProxy tunneling failed: Service UnavailableUnable to establish SSL connection.\n'
..
..
YYYY-MM-DDTHH:MM:SS PM UTC [47701]ERROR:vmware.appliance.update.update_b2b:
Got exception while trying discover at URL https://sddcfqdn/vmware/vcf/bundle/VMware/vRack/lcm/8745914e-695d-41b3-95f1-8fd049522036/a6a80d2e-171f-4f50-bc94-f34dab537697/iso/repos/patcher_repo: NotFound(messages=[{'id': 'com.vmware.appliance.update_error', 'default_message': '%s', 'args': ['Network failure. Check the network settings and try again.']}], data=None, error_type='NOT_FOUND') 'Traceback (most recent call last):\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1699, in _discoverUpdateAt\n copyFileFunc(path.join(manifestDir,\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 615, in wgetWrapper\n runCommandAndCheckResult(\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 403, in runCommandAndCheckResult\n raise exception\nvmware.appliance.update.update_functions.LocalizableException: {\'id\': \'com.vmware.appliance.update.network_error\', \'default_message\': \'Network error at target URL\', \'args\': []}\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 2047, in processURLUpdates\n header = _discoverUpdateAtUrl(url,\'latest\', enableListMajorUpgradeVersions)\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1915, in _discoverUpdateAtUrl\n return _discoverUpdateAt(manifestDir, packagesDir, copyFileFunc,\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1711, in _discoverUpdateAt\n vapiNotFound(messageListError("Network failure. Check the network settings and try again."))\n File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 170, in vapiNotFound\n raise ErrorFactory.new_not_found(messages=messages)\ncom.vmware.vapi.std.errors_provider.NotFound: {messages : [{\'id\': \'com.vmware.appliance.update_error\', \'default_message\': \'%s\', \'args\': [\'Network failure. Check the network settings and try again.\']}], data : None, error_type : NOT_FOUND}\n'

While checking on vCenter in /etc/sysconfig/proxy proxy config is enabled.

Resolution

Add SDDC Manager IP and FQDN entries in vCenter proxy config to bypass proxy server for connection.
SSH to vCenter server.
Take a backup of /etc/sysconfig/proxy in vCenter.
cp /etc/sysconfig/proxy /root/proxy.bak
Edit /etc/sysconfig/proxy file
Add SDDC Manager IP and FQDN for the parameter 'NO_PROXY' as like below,
NO_PROXY="localhost, 127.0.0.1, <SDDC_IP_ADDRESS>, <SDDC_FQDN>"
Reboot vCenter server.