NSX Federation standby Global Manager cluster still has expired self-signed certificates after running CARR script

Products

• VMware NSX

Issue/Introduction

A known issue impacting NSX 4.x version where self-signed certificates in NSX are expiring or expired. 

Certificate Analyzer, Results and Recovery (CARR) Script may have been used to resolve the expired self-signed certs issue. 

However, after running the CARR script against a Federated NSX environment, there might still be some self-signed certs not renewed in standby Global Manager cluster. 

This issue may not be exclusive to the standby GM site. It could also happen on active GM site or any local sites. This KB is to address the specific issue with standby Global Manager. 

The CARR script may show the replacement process completed successful. However, expired certs can still be seen in NSX UI such as the following example:

Environment

VMware NSX 

Cause

The CARR script did not replaced these certs due to not matching criteria for replacement. 

Resolution

You can replace the self-signed certificates for the following service types:

• MGMT_CLUSTER (aka VIP)
• CBM_CLUSTER_MANAGER
• K8S_MSG_CLIENT
• CBM_CORFU
• CCP
• APH_TN
• LOCAL_MANAGER
• GLOBAL_MANAGER
• APH (aka APH_AR)
• API
• WEB_PROXY
• IDPS-Reporting

In order to resolve this issue, new self-signed certificates need to be created for each of the expired self-signed certs then importing them and replacing the expired certs. 

1. SSH into a NSX manager node and issue the following command. 
   • openssl req -newkey rsa:2048 -extensions usr_cert -nodes -keyout <service-name-node-UUID>.key -x509 -days 36500 -out <service-name-node-UUID>.crt -subj "/CN=<service-type>" -sha256
     • -subj parameter should be used to match any existing cert details.
     • A typical NSX self-signed certificate has the following subject parameter "/C=US/ST=California/L=Palo Alto/O=VMware/CN=<SERVICE_TYPE>"
     • The Common Name (CN) for the certificate must match the expired cert it is replacing
2. Use the following commands against the newly created key-pair to fetch the key info in desired format for the next step.  
   1. awk 'NF {sub(//, ""); printf "%s\\n",$0;}' <service-name-node-UUID>.crt
   2. awk 'NF {sub(//, ""); printf "%s\\n",$0;}' <service-name-node-UUID>.key
3. Import the new cert using API call POST /api/v1/trust-management/certificates?action=import
   • Having the following in the body as json
     {
       "display_name": "<new-cert-name-can-be-the-same-as-the-expired-one>",
       "pem_encoded": "<content from 2.a from above>",
       "private_key": "<content from 2.b from above>"
     }
4. Verify that the new cert has been created in NSX UI and copy its UUID for next step
5. Apply the new cert to the service that has expired cert attached
   • POST /api/v1/trust-management/certificates/<New Certificate ID>?action=apply_certificate&service_type=<service-type>&node_id=<NSX-manager-node-UUID>
   • both service type and node ID used for the new certificate should match the expired one being replaced, if unsure, use the below API call to fetch the cert details:
     • GET /api/v1/trust-management/certificates/<Old Certificate ID>
     • An example of output:

       • {
             ....
             "used_by": [
                 {
                     "node_id": "<NSX-Manager-node-UUID>",
                     "service_types": [
                         "APH"
                     ]
                 }
             ],
             "resource_type": "certificate_self_signed",
             "id": "<Cert-UUID>",
             "display_name": "APH-AR certificate for node <NSX-Manager-node-UUID>",
         ...
         }

6. Verify that the new certificate has an increment on "where used" and the expired certificate has 0 on "where used".
7. The expired cert can then be deleted either through UI or using API call:
   • DELETE /api/v1/trust-management/certificates/<expired-cert-UUID>

Additional Information

If using VMware NSX 4.2, the certificate can also be replaced through UI