NSX Upgrade fails with "Prepare edge upgrade bundle failed on edge TransportNode"

Products

• VMware NSX

Issue/Introduction

• NSX Upgrade fails on NSX Edge Transport Node.
• NSX Upgrade UI reports: Edges upgrade has failed, check error details to determine if manual resolution is needed and 'Retry Upgrade'
• Error details is similar to

  Prepare edge upgrade bundle https://####/repository/####/Edge/nub/VMware-NSX-edge-####.nub failed on edge TransportNode ########-####-####-####-############: clientType EDGE , target edge fabric node id ########-####-####-####-############, return status Download and verify bundle failed with msg: Closing connection 0

• Log lines similar to the below are encountered on the NSX Manager in /var/log/upgrade-coordinator/upgrade-coordinator.log

  ERROR task-executor-4-1-workitem-EDGE-########-####-####-####-############ EdgeNodeUpgradeServiceImpl 941141 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30240" level="ERROR" subcomp="upgrade-coordinator"] Prepare edge upgrade bundle https://####/repository/####/Edge/nub/VMware-NSX-edge-####.nub failed on edge TransportNode ########-####-####-####-############: clientType EDGE , target edge fabric node id ########-####-####-####-############, return status Download and verify bundle failed with msg: Closing connection 0 .

• The FQDN or IP address in the URL mentioned by the message or log is one of the NSX Manager nodes.
• Inspecting the certificate applied to the API service of the NSX Manager node from the URL shows that the certificate does not contain the FQDN or IP address in Subject or Subject Alternative Name.
  For example, the certificate was created for the VIP FQDN only but was applied to the cluster (VIP) and all the individual NSX Managers.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX
VMware NSX-T Datacenter

Cause

If the certificate applied to the API service of all the NSX Manager nodes but that certificate was only created with the details of the VIP, then the certificate is actually invalid from the perspective of the API service of each NSX Manager node.
During the upgrade preparation, the NSX Edge node fails to retrieve the upgrade bundle because it rejects the invalid certificate presented by the NSX Manager node it contacts.

Resolution

This is a condition that may occur in a VMware NSX environment.

To implement a valid configuration, update the certificate applied to the NSX Manager API service with a certificate that is valid from its perspective:

• Either by using a wildcard certificate that is valid for the VIP FQDN and the individual NSX Manager FQDNs in a single certificate applied to all.
• Or include the VIP FQDN as Subject and NSX Manager FQDNs as Subject Alternative Names (SAN) in a single certificate applied to all.
  Starting in NSX 4.2.1, it is possible to create a CSR with SAN from the NSX UI.
  In earlier versions, the CSR should be created via API call (NSX API /api/v1/trust-management/csrs-extended) or via utilities else than NSX.
• Or create separate certificates for the VIP and each of the NSX Manager nodes.

This is also applicable if you use IP addresses instead of FQDNs, or need to declare both IP addresses and FQDN as SAN.