Unable to access NSX-T UI and API failure following a change in Manager Node certificates
search cancel

Unable to access NSX-T UI and API failure following a change in Manager Node certificates

book

Article ID: 312615

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • You have recently changed the NSX Manager node or cluster certificates
  • You are using third party certificates rather than the self signed NSX certificates which are created automatically when NSX is installed.
  • API calls executed against the nodes fail 
  • The following entries are observed in /var/log/proxy/envoy.log
[2024-03-22T14:24:52.685Z][22570][warning][config] [source/common/config/filesystem_subscription_impl.cc:43] Filesystem config update rejected: Error adding/updating listener(s) https-node-v4-local: Failed to load certificate chain from <inline>

Environment

VMware NSX
VMware NSX-T Data Center

Cause

There are multiple potential causes for this issue, but the most common are:

  • The PEM document has been imported incorrectly rather than as per the below example:
----- BEGIN CERTIFICATE -----
<leaf node certificate>
----- END CERTIFICATE-----
----- BEGIN CERTIFICATE -----
<intermediate ca certificate>
----- END CERTIFICATE-----
----- BEGIN CERTIFICATE -----
<root ca certificate>
----- END CERTIFICATE-----
  • The line separators in imported PEM file are DOS Style \r\n, which NSX does not correctly interpret. While NSX will allow the entry, the issue occurs when NSX eventually writes the PEM files for the associated service, they end with \n\r\n between each line, which isn't correctly parsed.

Resolution

A fix for PEM files with DOS-style newlines was made in NSX-T 4.1.1.

Workaround:

**It is recommended that backups are confirmed to be in place before making any changes**

Option 1) If the NSX-T Manager UI is not accessible and the API is not accessible for NSX-T Manager cluster then perform the below workaround:

Step 1 - Validate if the pem files are valid using below command if any one of the file has problem move to step 2:

openssl x509 -noout -text -in /home/secureall/secureall/.store/.tomcat_cert.pem
openssl x509 -noout -text -in /home/secureall/secureall/.store/.vip_cert.pem

Step 2 - Make a copy of the files ".tomcat_cert.pem" and ".vip_cert.pem"

cp /home/secureall/secureall/.store/.tomcat_cert.pem /tmp/.tomcat_cert.pem
cp /home/secureall/secureall/.store/.vip_cert.pem /tmp/.vip_cert.pem

Step 3 - Take a copy of the invalid pem files and run the following command from a helper Linux machine using "dos2unix", else rectify the issue manually if the error is noticeable:

dos2unix .tomcat_cert.pem
dos2unix .vip_cert.pem


Step 4 - Replace the rectified ".tomcat_cert.pem" and ".vip_cert.pem" at the path "/home/secureall/secureall/.store/"

Step 5 - Restart http service

restart service http


Option 2) If the NSX-T Manager UI is accessible and the API is accessible for NSX-T Manager cluster then perform the below workaround:

Step 1 - Re-import the CA-signed certificate.

  • Instead of choosing "Browse" and selecting the certificate PEM file, open the PEM file in a text editor like Notepad and copy-paste the contents into the "Certificate Contents" field.
  • Then, copy-paste the certificate's private key into the "Private Key" field. Be sure to un-select the "Service Certificate" option.
  • Once the certificate has been imported, note its ID - that will be needed later and will be referred to as "ca-certificate-id"
  • Log into any NSX manager node a root and run the following commands that will revert to the self-signed certificates created when the managers were installed.


Step 2 - Determine the node IDs of the three manager nodes.

  • These can be obtained from the System -> Appliances page in the NSX UI.
  • For each manager node, open the "Details" link and click on the clipboard icon next to "UUID" in the details page. Note the UUIDs of these nodes.
  • They will be referred to later as "nsx-mgr1-node_id", "nsx-mgr2-node_id", and "nsx-mgr3-node_id".


Step 3 - Apply the new certificate to the manager nodes and VIP.

  • Log into any manager node as root and run the following 4 commands:

curl -k -u admin -H "Content-Type: application/json" -X POST 'https://<nsx-mgr>/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=API&node_id=<nsx-mgr1-node_id>'

curl -k -u admin -H "Content-Type: application/json" -X POST 'https://<nsx-mgr>/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=API&node_id=<nsx-mgr2-node_id>'

curl -k -u admin -H "Content-Type: application/json" -X POST 'https://<nsx-mgr>/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=API&node_id=<nsx-mgr3-node_id>'

curl -k -u admin -H "Content-Type: application/json" -X POST 'https://<nsx-mgr>/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=MGMT_CLUSTER'


Step 4 - Restart the reverse proxy service on each manager node.

  • Log into each manager node as the admin user and run the comment "restart service http"


Once the correct certificate has been applied, you can delete the bad certificate from the System -> Certificates page in the NSX Manager UI.

Powered by Wolken Software