NSX edge upgrade fails with the error "download and verify bundle failed with msg: closing connection 5".

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• During the NSX Edge upgrade, the process is stuck at 1% for around 5 minutes and eventually fails with the error  "download and verify bundle failed with msg: closing connection 5".
  
  
• In the NSX Manager:

var/log/upgrade-coordinator/upgrade-coordinator.log:
ClientType EDGE, target edge fabric node id xxxxxxxx, return status Download and verify bundle failed with msg: Closing connection 5, canSkip: true
INFO http-nio-127.0.0.1-7442-exec-7 UpgradeQueryServiceImpl 644008 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coodinator"] Returning upgrade status summary for EDGE details as Prepare edge upgrade bundle https://<nsx-manager-fqdn>/repository/4.2.0.0.0.24105817/Edge/nub/VMWARE-NSX-edge-4.2.0.0.24105824.nub failed on edge TransportNode xxxxxxxxx: clientType EDGE, target edge fabric node id xxxxxxxx, return status Download and verify bundle failed with msg: Closing connection 5, ..

• In the edge:

/var/log/syslog.log:
Warning: Transient problem: Will retry 4 seconds. 3 retres left
WARNING: Failed to check connected manager and controller: Traceback
File /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper line 1490 in_validated_peer_cert_chain#012 cert_chain = get_peer_cert_chain(options, host, port)
Errno: Temporary failure in name resolution

• In some cases, we won't see the above DNS resolution error in the logs. In such case, on the problematic edge node, we can run <wget https://<nsx-manager-fqdn>/repository/4.1.2.4.0.23786733/Edge/nub/VMware-NSX-edge-4.1.2.4.0.23786751.nub> to try manually downloading the upgrade file from the NSX Manager and check if it succeeds. If it doesn't, that indicates a network connectivity issue between the NSX Manager and the Edge node.

• In the edge, nslookup does not resolve the FQDN and/or IP of NSX managers.

• NSX Manager is configured with dual stack and/or uses a CA certificate.

Environment

VMware NSX
VMware NSX-T Data Center

Cause

In this scenario, the NSX edge is unable to resolve the NSX Manager FQDN, which prevents it from downloading the bundle file. This issue arises when there is a dual stack and/or a CA certificate configured in the NSX Manager, causing it to use the FQDN instead of the IP address to communicate with the edges and transport nodes.

Having a proper hostname with a valid domain name is a requirement when there is a dual-stack and/or CA certificate  NSX Manager Installation Requirements.

Resolution

For this scenario, it is required to investigate why the edge is not able to connect to or resolve NSX manager FQDN or IP:

• Check if there is a firewall in between.
• Check if the DNS servers are available.
• Check if there is a wrong DNS configuration in the edge servers.
• Check if the Subnet Mask is Correct along with the CIDR Notation.
• Perform packet capture to analyze the packets.