Troubleshooting NSX Manager SSL Certificates

Products

VMware NSX

Issue/Introduction

This document describes the processes and resources needed for replacing SSL certificates for NSX-T

Environment

VMware NSX

Resolution

Determine which certs need to be replaced as well as requirements:

When a warning is seen about the certificates expiring, the status of all current certs can be found in System -> Certificates. Specifically look at the 'Validity', 'Expire Date' and 'Where Used' fields. If 'Where Used' is 0, then the certificate is not being used as a cluster or federation cert.

Here is a screenshot on NSX-T 4.x, showing the certificate view:

If the cert is being replaced for a reason other than expiring, detemine whether the environment is using federation and if just the manager certs or manager and federation certs are being replaced.

For a reference on the different types of certs available for replacement see Types of Certificates..

Determine the customer requirements:

Are self-signed SSL certs going to be used?
Do the certificates need to be signed by an external CA?
Do the certificates need to be externally generated? Common reasons would be
1. The need to add additional subject alternative names (SANS) to allow fewer certificates to be generated overall.
2. Using an existing wildcard cert.
3. Access to the cert and private key for import into a different system.
4. Generating the certificate for automatic trust through ADFS or other PKI system.
Is VMware Cloud Foundation (VCF) being utilized?

Generate and sign replacement certificates using the appropriate method determined by requirements:

Once the certs that need to be replaced have been determined and the requirements for those certs, follow the processes below to get the certificates generated and signed.

Self-signed certificates: If self-signed certificates are going to be utilized (question 1 is true and 2 and 3 above are false), then click "Create a Self-Signed Certificate" to automatically generate a self-signed certificate through the manager. Once completed the cert can be seen in the certificate store with '0' for the 'Where Used' column.
CA-signed certificates: If an external CA is used to sign their certificates and don't require access to the private key (Question 3 above is false) the following steps must be followed:
1. Generate a CSR (Certificate Signing Request) from the NSX-T manager by following "Create a Certificate Signing Request File".
2. Download the CSR and send it to the customer's CA for signing. Once signed, the CA will provide a copy of the signed cert as well as a chain of root certificates used by the CA.
3. Complete the CSR in the manager by importing the certificate using the following procedure "Import a Certificate for a CSR".
4. Import the CA trust bundle, which should have been provided by the CA in addition to the signed cert by using following procedure "Import or Update a Trusted CA Bundle"
Externally created certificates: If for any reason the need to generate the certificates outside of NSX-T follow the steps below for the generation.
1. After generating the certificate, if the it needs to be signed by an external CA, first generate a CSR request using Create a Certificate Signing Request File
2. Send the CSR to a CA for singing. Once signed, the CA will provide a copy of the singed cert as well as a chain of root certs used by the CA in return.
3. Import the signed certificate sent back from the CA as well as the private key used in generation by following "Import a Self-signed or CA-signed Certificate". Be sure to include the private key as well as the password used in generation.

Validate Certificates:

Now that certs are either imported or generated and signed, before assigning to a service, verify that NSX-T sees the certificates as valid by running the following API call:

GET https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=validate

To get the certificate ID, copy the 'ID' field seen in the manager GUI, or use the following API call to list all certificates and simply use the value of the 'id' field:

GET https://<nsx-mgr>/api/v1/trust-management/certificates

Replace generated certificates:

API call for replacing the certs are covered in "Replace Certificates"

If API Certificates are being replaced, utilize steps 1-5 of "Replace Certificates". Step 4 shows the API call for the local manager certificates (API cert) and Step 5 shows the API call for the overall cluster certificate.
If federation certificates are being replaced, utilize steps 1-3 and step 6 for the principal identity certificate and/or step 7 for the APR-APH certificate.

Additional Information

Supporting Documents:

For documentation on the different certificate types for federation and the managers see "Certificates for NSX Federation" and "Types of Certificates".

User-Imported CA-Signed Certificate is Expired - The CA-signed certificate that you imported to the NSX Manager appliance host has expired and you are unable to continue working with the NSX Application Platform.

Known Issues:

NSX alarms indicating certificates have expired or are expiring - certificate expires after 825 days in 3.2.1 - replace_cert.py script here to replace them automatically.

NSX Manager/NSX Global-Manager UI Not Accessible After Replacing CBM_* Certificates in 4.1.1 - after upgrade from 3.2.x to 4.1.1, when CBM_* certificate s replaced, UI is inaccessible.

Scripted process to Replace Expired or Self-signed VMware NSX-T Manager Certificates with VMCA-Signed Certificates - This KB helps to automate the entire process with a script.

NSX UI/API certificate is not in use on an NSX manager node - When viewing the System > Settings > Certificates page, you see that one of the API/UI certificates has a value of 0 in the Where Used column.

NSX Manager/NSX Global-Manager UI Not Accessible After Replacing CBM_* Certificates in 4.1.1 - This article provides the steps to bring up UI on one NSX manager node.

NSX-T Federation assigning Certificate to Local Manager fails with "Expected 1 local site certificate with service type LOCAL_MANAGER but found 0." - This issue occurs when the original local manager certificate was removed before the new one was applied.

NSX-T Federation Deployment: Connection from Local Manager to the Global Manager is broken in the NSX Federation after replacing the Local Managers mp-cluster certificate - The user sees that the "Appliances", "Latest Backup", "Automatic Backups" and "Remote Tunnel Endpoint Status" status on one/all Local Managers are "Not available"

VMware NSX Federation LOCAL_MANAGER or GLOBAL_MANAGER replaced expired certificate still alerting as expired - This is a known issue impacting VMware NSX.