vCenter server upgrade pre-check failed with the error "Network failure. Check the network settings and try again"
search cancel

vCenter server upgrade pre-check failed with the error "Network failure. Check the network settings and try again"

book

Article ID: 416494

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • When upgrading a VMware Cloud Foundation 5.2.1 environment to version 9, the vCenter upgrade precheck fails, displaying the error message: 'vCenter upgrade failed at the VCENTER_UPGRADE_PRECHECK stage.'
  • SDDC and NSX managers have been upgraded to version 9.
  • Reviewing /var/log/vmware/vcf/lcm/lcm-debug.log on the SDDC Manager, the following log excerpts were identified: 

    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,########################,2ba5] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-127.0.0.1-7400-exec-1] Upgrade error occured: vCenter upgrade failed at the VCENTER_UPGRADE_PRECHECK stage. vCenter upgrade failed. [Network failure. Check the network settings and try again.]
    Reference token ######
    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,########################,2ba5] [c.v.evo.sddc.lcm.model.task.Stage,http-nio-127.0.0.1-7400-exec-1] Stage details: Stage(id=32:3, name=null, description=null, type=VCENTER_UPGRADE_PRECHECK, order=3, startTime=null, endTime=null, duration=null, status=FAILED, errorType=null, errorCode=null, errorDescription=null, remediation=null, resources=null, errors=null, notifications=[Notification(severity=ERROR, stage=VCENTER_UPGRADE_PRECHECK, timestamp=########, message=LocalizableMessage(id=######-####-####-##########, defaultMessage=vCenter upgrade failed at the VCENTER_UPGRADE_PRECHECK stage. vCenter upgrade failed. [Network failure. Check the network settings and try again.], messageAttributes=null, localizedMessage=vCenter upgrade failed at the VCENTER_UPGRADE_PRECHECK stage. vCenter upgrade failed. [Network failure. Check the network settings and try again.], version=null), originator=VCF LCM Service, errorType=RECOVERABLE, remediationActions=[LocalizableMessage(id=########-#####-#####-########, defaultMessage=Please contact GSS for further assistance., messageAttributes=null, localizedMessage=Please contact GSS for further assistance., version=null)], referenceToken=######)], subtaskId=32, subtaskType=UPGRADE, taskId=#########-######-####-######, taskType=com.vmware.vcf.lcm.upgrade)
    YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,########################,0000,upgradeId=########-#####-#####-#######,resourceType=VCENTER,resourceId=######-####-####-####-######,bundleElementId=########-######-######-######] [c.v.e.s.l.p.c.u.VmwPrimitiveUtils,Upgrade-2] Exception occurred during vCenter upgrade:
    com.vmware.evo.sddc.lcm.model.error.LcmException: null
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.rdu.VlcmUpgrader.update(VlcmUpgrader.java:219)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.rdu.RduClientImpl.upgrade(RduClientImpl.java:317)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterRDUUpgrader.upgrade(VCenterRDUUpgrader.java:1171)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterRDUPrimitiveImpl.executeRduStages(VCenterRDUPrimitiveImpl.java:512)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterRDUPrimitiveImpl.executeUpgradeStages(VCenterRDUPrimitiveImpl.java:666)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterRDUPrimitiveImpl.upgradeHelper(VCenterRDUPrimitiveImpl.java:754)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterRDUPrimitiveImpl.postUpgradeWithState(VCenterRDUPrimitiveImpl.java:209)
            at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterRDUPrimitiveImpl.postUpgrade(VCenterRDUPrimitiveImpl.java:126)
            at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.postUpgradeAsync(PrimitiveServiceImpl.java:280)
            at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.lambda$postUpgrade$0(PrimitiveServiceImpl.java:152)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.base/java.lang.Thread.run(Thread.java:840)
    Caused by: com.vmware.vapi.std.errors.NotFound: NotFound (com.vmware.vapi.std.errors.not_found) => {
        messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
        id = com.vmware.appliance.update_error,
        defaultMessage = %s,
        args = [Network failure. Check the network settings and try again.],
        params = <null>,
        localized = <null>
    }],
        data = <null>,
        errorType = NOT_FOUND
    }

  • The SDDC Manager successfully invoked the appropriate vCenter APIs to initiate the upgrade; however, the process encountered an error or exception. This was confirmed by reviewing /var/log/vmware/applmgmt/applmgmt.log on the vCenter server.

    YYYY-MM-DDTHH:MM:SS [9215]DEBUG:vmware.appliance.update.update_functions:Running /usr/bin/wget --server-response --tries 3 --waitretry 1 --connect-timeout 10 -r -np -nH -nd -A  -P /storage/core/software-update/tmp/latest https://SDDC_Manager_FQDN/vmware/vcf/bundle/VMware/vRack/lcm/######-####-#####-####/######-####-####-####-######/iso/repos/patcher_repo/manifest/manifest-latest.xml -e CENSORED -e CENSORED
    YYYY-MM-DDTHH:MM:SS [9215]DEBUG:vmware.appliance.update.update_functions:runCommandAndCheckResult failed: '--YYYY-MM-DDTHH:MM:SS--  https://SDDC_Manager_fqdn/vmware/vcf/bundle/VMware/vRack/lcm/######-####-####-####-######/######-####-####-####-######/iso/repos/patcher_repo/manifest/manifest-latest.xml\nResolving Proxy_server_FQDN... Proxy_server_IP\nConnecting to Proxy_server_FQDN|Proxy_server_IP|:Proxy_port... connected.\nProxy tunneling failed: ForbiddenUnable to establish SSL connection.\n'
    YYYY-MM-DDTHH:MM:SS [9215]ERROR:vmware.appliance.update.update_b2b:
    Got exception while trying discover at URL https://SDDC_Manager_fqdn/vmware/vcf/bundle/VMware/vRack/lcm/######-####-####-####-######/######-####-####-####-######/iso/repos/patcher_repo: NotFound(messages=[{'id': 'com.vmware.appliance.update_error', 'default_message': '%s', 'args': ['Network failure. Check the network settings and try again.']}], data=None, error_type='NOT_FOUND') 'Traceback (most recent call last):\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1699, in _discoverUpdateAt\n    copyFileFunc(path.join(manifestDir,\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 615, in wgetWrapper\n    runCommandAndCheckResult(\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 403, in runCommandAndCheckResult\n    raise exception\nvmware.appliance.update.update_functions.LocalizableException: {\'id\': \'com.vmware.appliance.update.network_error\', \'default_message\': \'Network error at target URL\', \'args\': []}\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 2047, in processURLUpdates\n    header = _discoverUpdateAtUrl(url,\'latest\', enableListMajorUpgradeVersions)\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1915, in _discoverUpdateAtUrl\n    return _discoverUpdateAt(manifestDir, packagesDir, copyFileFunc,\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1711, in _discoverUpdateAt\n    vapiNotFound(messageListError("Network failure. Check the network settings and try again."))\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 170, in vapiNotFound\n    raise ErrorFactory.new_not_found(messages=messages)\ncom.vmware.vapi.std.errors_provider.NotFound: {messages : [{\'id\': \'com.vmware.appliance.update_error\', \'default_message\': \'%s\', \'args\': [\'Network failure. Check the network settings and try again.\']}], data : None, error_type : NOT_FOUND}\n'
    YYYY-MM-DDTHH:MM:SS PM EDT [9215]ERROR:vmware.appliance.update.update_b2b:Got Exception during discover updates {messages : [{'id': 'com.vmware.appliance.update_error', 'default_message': '%s', 'args': ['Network failure. Check the network settings and try again.']}], data : None, error_type : NOT_FOUND}

Environment

VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

Network traffic is being blocked by a proxy server configured on the vCenter server, which prevents the SDDC manager from configuring the repository settings in vCenter during the upgrade.

Resolution

Unblock the network traffic between the SDDC manager and vCenter server to resolve the issue.

Workaround:

1) Temporarily disable the vCenter proxy configuration from the VAMI page.
2) If the proxy cannot be disabled, add the SDDC manager fqdn to the no proxy list of the vCenter server.

Additional Information


vCenter Server upgrade failed with error "VCENTER UPGRADE SET REPO" from SDDC Manager UI

Powered by Wolken Software