Troubleshooting NSX Manager SSL Certificates

Products

• VMware NSX

Issue/Introduction

This article explains how to replace the SSL certificates used by NSX-T (NSX Manager) and links to the documentation you need for each method.

Environment

VMware NSX

Resolution

Determine which certificates need to be replaced, and your requirements

If you see a warning that certificates are expiring, check the status of all current certificates in System → Certificates. Look at the Validity, Expire Date, and Where Used fields. If Where Used is 0, that certificate is not in use as a cluster or federation certificate.

Here is a screenshot on NSX-T 4.x showing the certificate view:

(NSX-T 4.x System → Certificates page, showing the Validity, Expire Date, and Where Used columns.)

If you're replacing a certificate for a reason other than expiry, first determine whether your environment uses federation, and whether you're replacing only the manager certificates or both the manager and federation certificates.

For details on the certificate types available for replacement, see Types of Certificates.

Determine your requirements:

1. Will you use self-signed SSL certificates?
2. Do the certificates need to be signed by an external CA (certificate authority)?
3. Do the certificates need to be created outside NSX-T? Common reasons:
   1. You need to add extra Subject Alternative Names (SANs) so fewer certificates are needed overall.
   2. You want to use an existing wildcard certificate.
   3. You need access to the certificate and private key to import into another system.
   4. You want the certificate to be automatically trusted through ADFS or another PKI system.
4. Are you using VMware Cloud Foundation (VCF)?

Generate and sign the replacement certificates using the method that matches your requirements:

Once you've identified which certificates to replace and your requirements for them, follow the matching process below to generate and sign them.

• Self-signed certificates - If you're using self-signed certificates (question 1 is yes, and 2 and 3 are no:
  • Follow Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX to use the CARR script to replace all self-signed cluster certificates that are expired or expiring. 
  • Or to use the NSX UI, follow Create a Self-Signed Certificate to have NSX Manager generate one automatically. When it's done, the new certificate appears in the certificate store with 0 in the Where Used column.
• CA-signed certificates: If an external CA will sign your certificates and you don't need access to the private key (question 3 is no), follow these steps:
  • Generate a CSR (Certificate Signing Request) from NSX Manager by following Create a Certificate Signing Request File.
  • Download the CSR and send it to your CA for signing. The CA returns the signed certificate along with the chain of root certificates it uses.
  • Import the signed certificate back into NSX Manager to complete the CSR, using Import a Certificate for a CSR.
  • Import the CA trust bundle (provided by the CA alongside the signed certificate) using Import or Update a Trusted CA Bundle.
• Externally created certificates: If you need to generate the certificates outside NSX-T for any reason, follow these steps:
  •  After generating the certificate, if it needs to be signed by an external CA, first generate a CSR using Create a Certificate Signing Request File.
  • Send the CSR to a CA for signing. The CA returns the signed certificate along with the chain of root certificates it uses.
  • Import the signed certificate (returned by the CA) together with the private key used to generate it, by following Import a Self-signed or CA-signed Certificate. Be sure to include the private key and the password used during generation.

Validate Certificates:

Now that the certificates are imported or generated and signed, verify that NSX-T sees them as valid before assigning them to a service. Run the following API call:

GET https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=validate

To find the certificate ID, copy the ID field shown in the NSX Manager UI, or run the following API call to list all certificates and use the value of its id field:

GET https://<nsx-mgr>/api/v1/trust-management/certificates

Replace generated certificates:

Before NSX 4.2: API call for replacing the certs are covered in Replace Certificates.

• If you're replacing API certificates, use steps 1-5 of Replace Certificates. Step 4 is the API call for the Manager API certificate and step 5 is for the overall shows the API call for the overall cluster VIP certificate. 
• If you're replacing NSX Federation certificates, use steps 1-3, then step 6 for the principal identity certificate and/or step 7 for the APR-APH certificate.

NSX 4.2 and above:

• Replace Certificates Through NSX Manager
• Replace Certificates Through API
  
•  

Additional Information

Supporting Documents:

For documentation on the different certificate types for federation and the managers see Certificates for NSX Federation and Types of Certificates. 

• User-Imported CA-Signed Certificate is Expired - The CA-signed certificate that you imported to the NSX Manager appliance host has expired and you are unable to continue working with the NSX Application Platform.

 

Known Issues:

• Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX - The recommended method to check for certificate related issues and regenerate new self-signed certificates in one shot is the CARR script in this KB.
• NSX alarms indicating certificates have expired or are expiring - certificate expires after 825 days in 3.2.1 - replace_cert.py script here to replace them automatically.
• NSX Manager/NSX Global-Manager UI Not Accessible After Replacing CBM_* Certificates in 4.1.1 - after upgrade from 3.2.x to 4.1.1, when CBM_* certificate s replaced, UI is inaccessible.
• Scripted process to Replace Expired or Self-signed VMware NSX-T Manager Certificates with VMCA-Signed Certificates - This KB helps to automate the entire process with a script.
• NSX UI/API certificate is not in use on an NSX manager node  - When viewing the System > Settings > Certificates page, you see that one of the API/UI certificates has a value of 0 in the Where Used column.
• NSX Manager/NSX Global-Manager UI Not Accessible After Replacing CBM_* Certificates in 4.1.1  - This article provides the steps to bring up UI on one NSX manager node. 
• NSX-T Federation assigning Certificate to Local Manager fails with "Expected 1 local site certificate with service type LOCAL_MANAGER but found 0."  - This issue occurs when the original local manager certificate was removed before the new one was applied.
• NSX-T Federation Deployment: Connection from Local Manager to the Global Manager is broken in the NSX Federation after replacing the Local Managers mp-cluster certificate - The user sees that the "Appliances", "Latest Backup", "Automatic Backups" and "Remote Tunnel Endpoint Status" status on one/all Local Managers are "Not available"
• VMware NSX Federation LOCAL_MANAGER or GLOBAL_MANAGER replaced expired certificate still alerting as expired - This is a known issue impacting VMware NSX.