NSX Edge Node upgrade fails with the error "download and verify bundle failed with msg: closing connection 5".

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• During the NSX Edge Node upgrade, the process is stuck at 1% for around 5 minutes and eventually fails with the error.  "download and verify bundle failed with msg: closing connection 5".
  
  
• In the NSX Manager, entries similar to the ones  below may be visible in var/log/upgrade-coordinator/upgrade-coordinator.log:
  

  ClientType EDGE, target edge fabric node id #########, return status Download and verify bundle failed with msg: Closing connection 5, canSkip: true
  
  INFO http-nio-127.0.0.1-7442-exec-7 UpgradeQueryServiceImpl 644008 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coodinator"] Returning upgrade status summary for EDGE details as Prepare edge upgrade bundle https://<nsx-manager-fqdn>/repository/4.2.0.0.0.24105817/Edge/nub/VMWARE-NSX-edge-4.2.0.0.24105824.nub failed on edge TransportNode #########: clientType EDGE, target edge fabric node id #########, return status Download and verify bundle failed with msg: Closing connection 5, ..

• In the Edge Node, entries similar to the ones below may be encountered in /var/log/syslog.log: 
  

  Warning: Transient problem: Will retry 4 seconds. 3 retres left
  WARNING: Failed to check connected manager and controller: Traceback
  File /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper line 1490 in_validated_peer_cert_chain#012 cert_chain = get_peer_cert_chain(options, host, port)
  Errno: Temporary failure in name resolution

• In some cases, we won't see the above DNS resolution error in the logs. In such case, on the problematic edge node, we can run <wget https://<nsx-manager-fqdn>/repository/4.1.2.4.0.23786733/Edge/nub/VMware-NSX-edge-4.1.2.4.0.23786751.nub> to try manually downloading the upgrade file from the NSX Manager and check if it succeeds. If it doesn't, it indicates a network connectivity issue between the NSX Manager and the Edge node.

• In the Edge Node, nslookup does not resolve the FQDN and/or IP of NSX managers.

• NSX Manager is configured with dual stack and/or uses a CA certificate.

Environment

VMware NSX
VMware NSX-T Data Center

Cause

In this scenario, the NSX edge is unable to resolve the NSX Manager FQDN, which prevents it from downloading the bundle file. This issue arises when there is a dual-stack and/or a CA certificate configured in the NSX Manager, causing it to use the FQDN instead of the IP address to communicate with the edges and transport nodes.

Having a proper hostname with a valid domain name is a requirement when there is a dual-stack and/or CA certificate NSX Manager Installation Requirements.

Alternative scenario, the NSX edge is unable to connect to the NSX Managers over port 443 to download the node upgrade bundle due to a firewall in between the two devices.

Resolution

This is a condition that may occur in a VMware NSX environment.

For this scenario, it is required to investigate why the edge is not able to connect to or resolve NSX manager FQDN or IP:

• Check if there is a firewall in between.
  • Run a netcat test from root mode of the NSX Edge Node in question: nc -zv <nsx-manager-IP> 443
  • https://ports.broadcom.com/home/NSX - port 443 from NSX Transport Node to NSX Manager is needed for install/upgrade operations
• Check if the DNS servers are available.
• Check if there is a wrong DNS configuration in the edge servers.
• Check if the Subnet Mask is Correct along with the CIDR Notation.
• Perform packet capture to analyze the packets.

Additional Information

If the issue is related to DNS, as a temporary workaround to complete the upgrade, populate the NSX manager DNS information in the /etc/hosts file to complete the upgrade.

<NSX Manager IP> <NSX Manager FQDN> <NSX Manager short name>

##.##.##.## nsx-manager.example.com nsx-manager

Later, resolve the DNS issue.