Failed with return-code 1 (Error occurred while executing download_lcp script: LCP bundle download failed with return-code 60

Products

• VMware NSX

Issue/Introduction

- During the NSX Host upgrade, the transport nodes fails with error: Hosts upgrade has failed, check error details to determine if manual resolution is needed and 'Retry Upgrade'

- From ESXi host (Transport Node) logs:

/var/run/log/nsx-syslog.log

81387:2024-09-08T13:30:19.979Z Wa(180) nsx-sfhc[2100497]: NSX 2100497 - [nsx@6876 comp="nsx-esx" subcomp="nsxsfhc" tid="2102267" level="WARNING"] Command python -B -c "from __future__ import print_function; import os; os.chdir('/tmp/'); import download_lcp; print(download_lcp.fetch('https://x.x.x.x/repository/4.1.2.4.0.23786733/HostComponents/esx80/nsx-lcp-4.1.2.4.0.23786738-esx80.zip','nsx-lcp-4.1.2.4.0.23786738-esx80.zip','e78aead6b60-####-####-####-##########'))" failed with return-code 1 (Error occurred while executing download_lcp script: LCP bundle download failed with return-code 60

Environment

VMware NSX-T Data Center

VMware NSX

Cause

- From ESXi/Transport Node logs, we could see the issue is related to certs expired on NSX causing the issue:

var/run/log/nsx-syslog.log

81385:2024-09-08T13:30:19Z In(182) nsx-logger: NSX 91903243 - [nsx@6876 comp="nsx-esx" subcomp="curl_wrapper" username="root" level="INFO"] certificate verification e78aead6b60-####-####-####-########## from x.x.x.x:443 failed: certificate has expired

- There are tomcat certificate and mp-cluster certificates expired on all the NSX manager nodes and they need to be replaced.

Resolution

Check NSX UI to identify if any certificates have expired.( NSX UI > System > Certificates)

Option 1: No certificate is expired

This issue is caused by curl wrapper issue , please apply the workaround provided in the article linked below:

https://knowledge.broadcom.com/external/article?articleId=324178

 

Option 2: Certificates have expired

If certificates have expired, please proceed with the workaround provided below:

- To replace the tomcat and Management cluster certificates following are the APIs:

--> MP Cluster certificate is one cert for the whole NSX cluster

--> Tomcat certificate is one per manager node

To replace MP cluster certificate:

1. Create a self signed certificate: https://docs.vmware.com/en/VMware-NSX/4.1/administration/GUID-9BBF8A54-DFBD-4B24-B7A1-492CB42DD0D5.html

2. Validate the certificate: GET https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=validate

3. To replace the certificate of manager node (tomcat) use the following API call: POST /api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=API&node_id=<node-id>

(Perform the above 3 steps for the other 2 manager nodes)

To replace MP cluster certificate:

1. Create a self signed certificate: https://docs.vmware.com/en/VMware-NSX/4.1/administration/GUID-9BBF8A54-DFBD-4B24-B7A1-492CB42DD0D5.html

2. Validate the certificate: GET https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=validate

3. To replace the certificate of manager node (tomcat) use the following API call: POST /api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=MGMT_CLUSTER

Once the expired certificates are replaced, Hosts upgrades will complete with no errors.

Reference: https://docs.vmware.com/en/VMware-NSX/4.1/administration/GUID-50C36862-A29D-48FA-8CE7-697E64E10E37.html

Additional Information

Tomcat

This is an API certificate used for external communication with individual NSX Manager nodes through UI or API.

mp-cluster

This is an API certificate used for external communication with the NSX Manager cluster using the cluster VIP, through UI or API.