"Network failure. Check the network settings and try again." VC patch fails with error under Updates tab in VAMI

Products

• VMware vCenter Server

Issue/Introduction

• Unable to patch VC from VAMI (vCenter Server Appliance Management Interface), under the updates tab, the below error message is seen:
  "Network failure. Check the network settings and try again".
  
  
• The download URL is updated as per the new changes. Refer: VCF authenticated downloads configuration update instructions
  
  Example:
   
  • 8.x
    • https://dl.broadcom.com/<downloadtoken>/PROD/COMP/VCENTER/vmw/8d167796-34d5-4899-be0a-6daade4005a3/8.0.3.00400/
  • 7.x
    • https://dl.broadcom.com/<downloadtoken>/PROD/COMP/VCENTER/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.3.00800/
      
      
• On the vCenter Server, in /var/log/vmware/applmgmt/applmgmt.log


  YYYY-MM-DDTHH:MM:SS [PID]DEBUG:vmware.appliance.update.update_functions:runCommandAndCheckResult failed: '--YYYY-MM-DD HH:MM:SS-- https://dl.broadcom.com/<Token>/PROD/COMP/VCENTER/vmw/8d167796-34d5-4899-be0a-6daade4005a3/8.0.3.00400/manifest/manifest-latest.xml\nResolving dl.broadcom.com... ###.###.###.###, ###.###.###.###, ####:####:#::##, ...\nConnecting to dl.broadcom.com|###.###.###.###|:443... connected.\nUnable to establish SSL connection.\n'

  YYYY-MM-DDTHH:MM:SS [PID]ERROR:vmware.appliance.update.update_b2b: Got exception while trying discover at URL https://dl.broadcom.com/<Token>/PROD/COMP/VCENTER/vmw/8d167796-34d5-4899-be0a-6daade4005a3/8.0.3.00400: NotFound(messages=[{'id': 'com.vmware.appliance.update_error', 'default_message': '%s', 'args': ['Network failure. Check the network settings and try again.']}], data=None, error_type='NOT_FOUND') 'Traceback (most recent call last):\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1699, in _discoverUpdateAt\n    copyFileFunc(path.join(manifestDir,\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 615, in wgetWrapper\n    runCommandAndCheckResult(\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 403, in runCommandAndCheckResult\n    raise exception\nvmware.appliance.update.update_functions.LocalizableException: {\'id\': \'com.vmware.appliance.update.network_error\', \'default_message\': \'Network error at target URL\', \'args\': []}\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 2047, in processURLUpdates\n    header = _discoverUpdateAtUrl(url,\'latest\', enableListMajorUpgradeVersions)\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1915, in _discoverUpdateAtUrl\n    return _discoverUpdateAt(manifestDir, packagesDir, copyFileFunc,\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1711, in _discoverUpdateAt\n    vapiNotFound(messageListError("Network failure. Check the network settings and try again."))\n  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 170, in vapiNotFound\n    raise ErrorFactory.new_not_found(messages=messages)\ncom.vmware.vapi.std.errors_provider.NotFound: {messages : [{\'id\': \'com.vmware.appliance.update_error\', \'default_message\': \'%s\', \'args\': [\'Network failure. Check the network settings and try again.\']}], data : None, error_type : NOT_FOUND}\n'

  YYYY-MM-DDTHH:MM:SS [PID]ERROR:vmware.appliance.update.update_b2b:Got Exception during discover updates {messages : [{'id': 'com.vmware.appliance.update_error', 'default_message': '%s', 'args': ['Network failure. Check the network settings and try again.']}], data : None, error_type : NOT_FOUND} :
YYYY-MM-DDTHH:MM:SS [PID]INFO:vmware.appliance.update.update_pending:No updates found

• The same issue may be encountered when the Proxy configuration is changed. The below command can confirm if the vCenter is trying to connect to the old Proxy Server:
   
  • curl -v http://<Proxy_FQDN>:<portNumber>

Environment

• vCenter Server 8.x
• vCenter Server 7.x

Cause

• This issue is observed when the connection between vCenter Server and "dl.broadcom.com" is blocked by the network firewalls.
• The issue will occur by network-level traffic filtering or monitoring performed by a firewall within the customer environment.
• Follow any of below steps to confirm the connectivity
  
  
  • • Test patch manifest file download using 'wget' :
      
      • SSH to the vCenter Server and login with 'root' credentials.
      • Execute 'wget' command to download the patch manifest file. Copy the URL from the Updates Settings on VAMI.
        Example: wget https://dl.broadcom.com/<Download Token>/PROD/COMP/VCENTER/vmw/8d167796-34d5-4899-be0a-6daade4005a3/<version>/manifest/manifest-latest.xml

wget https://dl.broadcom.com/<Download Token>/PROD/COMP/VCENTER/vmw/8d167796-34d5-4899-be0a-6daade4005a3/8.0.3.00600/manifest/manifest-latest.xml
        
        Note:
        • Replace <Download Token> with the token generated from the support portal in above wget command.
        • Replace <version> from this KB article to patch VC to the latest available patch.
          
          
    • Test SSL handshake using 'openssl' command:
      1. SSH to the vCenter Server and login with 'root' credentials
      2. Execute below openssl command to perform SSL handshake (client and server hello)
         
         openssl s_client -connect dl.broadcom.com:443

Resolution

• Allow / whitelist the connection from vCenter Server to dl.broadcom.com in the Firewall.
• Ensure port 443 between the vCenter Server and dl.broadcom.com is not blocked.
• This behavior can also be influenced by AV filtering, Deep packet inspection (DPI) or SSL inspection features on certain firewall platforms. Bypassing these inspections or allowing direct access to the update servers may be required.

Additional Information

Manually update the patch URL with the download token refer this article: Unable to patch the vCenter via VAMI as it fails to download the updates from Broadcom public repositories 

Downloading the token from the Support Portal refer this article: VCF authenticated downloads configuration update instructions