Upgrading or deploying an VMware NSX Edge node or Manager appliance from NSX manager UI fails. Errors related to "curl

Products

VMware NSX VMware NSX-T Data Center

Issue/Introduction

When trying to deploy a VMware NSX 4.1.x Manager or Edge node from the UI it fails with the error:

Error while fetching ovf file. ASN length at position 2 curl_wrapper: (60) SSL: no alternative certificate subject name matches target host name '<nsx-manager-hostname>'

NSX Manager log /var/log/syslog has the following entries:

2023-04-28T12:49:01.517Z <nsx-manager-fqdn> NSX 4541 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP31705" level="ERROR" subcomp="manager"] For [test], error: Error while fetching ovf file. ASN length at position 2#012curl_wrapper: (60) SSL: no alternative certificate subject name matches target host name '<nsx-manager-fqdn>'#012

Deploying an Manager or Edge node via OVF in vCenter does not encounter the same issue.
This issue can also be seen during upgrade of a VMware NSX Edge or Manager appliance.
An error similar to the following may be seen during upgrade-coordinator upgrade while the NSX Manager repositories are being synced:

[<IP>] Unable to connect to File /repository/<version.build>/Manager/vmware-mount/libgobject-2.0.so.0 on source <nsx-manager-fqdn>. Please verify file exists on source and install-upgrade service is up.
NSX Manager log /var/log/proton/nsxapi.log has the following entries:

INFO RepoSyncThread-1687161993610 RepoSyncFileHelper 95373 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Command to get server info for https://nsxt.example.com:443/repository/4.1.X/<path_to_file> returned result CommandResultImpl [commandName=null, pid=3022439, status=FAILED, errorCode=60, errorMessage=Unexpected ASN length at position 2
curl_wrapper: (60) SSL: no alternative certificate subject name matches target host name 'nsxt'

OR

INFO RepoSyncThread-1695020706074 RepoSyncFileHelper 4977 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Command to check if remote file exists for https://nsxt.example.com:443/repository/4.1.X/<path_to_file> returned result CommandResultImpl [commandName=null, pid=1406936, status=SUCCESS, errorCode=0, errorMessage=null, commandOutput=Unexpected DNS name at position 78

OR

INFO RepoSyncThread-1698231201309 RepoSyncFileHelper 2664864 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Command to check if remote file exists for https://nsxt.example.com:443/4.1.X/<path_to_file> returned result CommandResultImpl [commandName=null, pid=3775111, status=FAILED, errorCode=51, errorMessage=curl_wrapper: (51) SSL: no alternative certificate subject name matches target host name 'nsxt-fqdn.com', commandOutput=null]

There may also be errors related to 'curl_wrapper'

TransportNode c#####-####-####-####-########7552: clientType EDGE , target edge fabric node id c#####-####-####-####-########7552, return status download_os execution failed with msg: Exception during OS download: Command ['/usr/bin/python3', '/opt/vmware/nsx-common/python/nsx_utils/curl_wrapper', '--show-error', '--retry', '6', '--output', '/image/VMware-NSX-edge-4.2.0.1.0.24210175/files/target.vmdk', '--thumbprint', '2###################################################################1', 'https://nsx-mngr/repository/4.2.0.1.0.24210154/Edge/ovf/nsx-edge.vmdk'] returned non-zero code 28: b'curl_wrapper: (28) Failed to connect to <nsx-mngr> port 443: Connection timed out'

Note:
The NSX version in the above log entry may be any 4.1.X version.
The manager FQDN, could also be an IP address.

Environment

VMware NSX 4.1.X

Cause

There is a software issue with the "curl_wrapper" utility which is used in communication between TNs and Manager nodes.

Resolution

This issue is resolved in VMware NSX 4.2.0 available at Broadcom Downloads.

Workaround:
If the issue encountered is when an Edge or Manager node is being deployed from the NSX UI, as a workaround the appliances can instead be deployed from the vSphere Client using an OVF file and then join it to the management plane. Please refer to the following document for further information: VMware NSX Installation Guide.

If the issue is encountered during upgrade or it is preferred to deploy appliances via the NSX UI, the following workaround can be used instead.

Warning: The following procedure involves changing files on NSX appliances, if implemented incorrectly this can be destructive. Ensure latest backups are present and the passphrase is known.

Download the attached curl_wrapper file and scp it to the /tmp directory on the 3 NSX Manager and the Edge nodes.

On each appliance:

1) Backup the existing curl wrapper file:

# cp /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper.bak

2) Apply the patch:

# cp /tmp/curl_wrapper /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper

3) Proceed with install or upgrade, no service restart is required.

Additional Information

Note there are other issues with similar symptoms that should be reviewed and ruled out

- Upgrading an NSX manager appliance from NSX manager UI fails and repo sync is in failed status
- Deploying a NSX Manager node fails with error: certificate validation failed "no alternative certificate subject name matches target host name".

Attachments

curl_wrapper get_app