Change NSX manager Certificates, in an SDDC environment with multiple SANs and no Short Name FQDNs

Products

• VMware SDDC Manager
• VMware NSX-T Data Center

Issue/Introduction

SDDC CSR creation for NSX Cluster, generates a Certificate Signing request, with a short name, but there is no short name selected or added, when generating the CSR.

NSX VIP + 3 nodes CSR creation from SDDC, generates a CSR with a short-name.
Behavior consistent, usually impacts the management Domain, furthermore some Certificate Authorities see this, as a that as a security risk. 

Request is to generate certificate for 
- NSX_VIP + NSX01_FQDN + NSX02_FQDN + NSX03_FQDN  
CSR result will read out 
- NSX_VIP + NSX01_short_name + NSX01_FQDN + NSX02_FQDN + NSX03_FQDN 

root@SDDC_Manager [ / ]# openssl req -in test.csr -noout -text | grep -i -A1 "X509v3 Subject Alternative Name:"
            X509v3 Subject Alternative Name:
                DNS:<NSX01_short_name>, DNS:<NSX_VIP>, DNS:<NSX01_FQDN >, DNS:<NSX02_FQDN>, DNS:<NSX03_FQDN>

Environment

SDDC Version: 4.5.0.0 Build 20612863 to SDDC Version: 5.1.1.0Build 23480823

Resolved in 5.2.0.0Build 24108943

Cause

- We use the NSXT API to fetch the deployment node details and we use that information to update the SAN names.

GET /api/v1/cluster/nodes/deployments
{
  "results" : [ ],
  "result_count" : 0
}

- The API returns request information for every attempted deployment of a cluster node VM.
- Only the auto-deployed nodes give output for this.

 

Resolution

Resolution:
- resolved in 5.2.0.0Build 24108943

Workaround:
- The workaround means replacing the certificates via NSX UI and pushing said certificates via API calls.
- The following API calls ca be run on the NSX Nodes, or on the SDDC manager, in order to obtain the Certificate Signing Request without the short-name.
- all values between "<>" are variables that need to be changed, Country, key Size, Common Name and so on.

NSX_VIP
curl -H "Content-Type: application/json" -X POST -d '{"algorithm":"RSA","key_size":"<3072>","subject":{"attributes":[{"key":"CN","value":"<NSX_VIP>"},{"key":"O","value":"<VMWare Inc.>"},{"key":"OU","value":"<VMware IT department>"},{"key":"C","value":"<US>"},{"key":"ST","value":"<California>"},{"key":"L","value":"<Palo Alto>"}]},"extensions":{"subject_alt_names":{"dns_names":["NSX_VIP","NSX01_FQDN","NSX02_FQDN,"NSX03_FQDN"]}}}' https://<NSX_VIP>/api/v1/trust-management/csrs-extended -k -u "<admin> :<admin_password>" > NSX_VIP.csr

NSX01_FQDN
curl -H "Content-Type: application/json" -X POST -d '{"algorithm":"RSA","key_size":"<3072>","subject":{"attributes":[{"key":"CN","value":"<NSX_VIP>"},{"key":"O","value":"<VMWare Inc.>"},{"key":"OU","value":"<VMware IT department>"},{"key":"C","value":"<US>"},{"key":"ST","value":"<California>"},{"key":"L","value":"<Palo Alto>"}]},"extensions":{"subject_alt_names":{"dns_names":["NSX_VIP","NSX01_FQDN","NSX02_FQDN,"NSX03_FQDN"]}}}' https://<NSX01_FQDN>/api/v1/trust-management/csrs-extended -k -u "<admin> :<admin_password>" > NSX01_FQDN.csr

Repeat for each Node VIP
NSX02_FQDN
curl -H "Content-Type: application/json" -X POST -d '{"algorithm":"RSA","key_size":"<3072>","subject":{"attributes":[{"key":"CN","value":"<NSX_VIP>"},{"key":"O","value":"<VMWare Inc.>"},{"key":"OU","value":"<VMware IT department>"},{"key":"C","value":"<US>"},{"key":"ST","value":"<California>"},{"key":"L","value":"<Palo Alto>"}]},"extensions":{"subject_alt_names":{"dns_names":["NSX_VIP","NSX01_FQDN","NSX02_FQDN,"NSX03_FQDN"]}}}' https://<NSX02_FQDN>/api/v1/trust-management/csrs-extended -k -u "<admin> :<admin_password>" > NSX02_FQDN.csr

NSX03_FQDN
curl -H "Content-Type: application/json" -X POST -d '{"algorithm":"RSA","key_size":"<3072>","subject":{"attributes":[{"key":"CN","value":"<NSX_VIP>"},{"key":"O","value":"<VMWare Inc.>"},{"key":"OU","value":"<VMware IT department>"},{"key":"C","value":"<US>"},{"key":"ST","value":"<California>"},{"key":"L","value":"<Palo Alto>"}]},"extensions":{"subject_alt_names":{"dns_names":["NSX_VIP","NSX01_FQDN","NSX02_FQDN,"NSX03_FQDN"]}}}' https://<NSX03_FQDN>/api/v1/trust-management/csrs-extended -k -u "<admin> :<admin_password>" > NSX03_FQDN.csr

 

1. Obtain the files, either all at once, or one by one and do a rolling replacement.
   I found it easier to run the rolling replacement - no, no reboot is needed
   
   
2. Once the CSR files are obtained we need to extract the relevant information.
   - two main things, are needed from the generated CSR files - the CSR string itself and the "id"/ "display_name"
   
   root@NSX02_FQDN:~# cat NSX02_FQDN.csr
{
  "subject" : {
    "attributes" : [ {
      "key" : "CN",
      "value" : "<NSX_VIP>"
    }, {
      "key" : "O",
      "value" : "<VMWare Inc.>"
    }, {
      "key" : "OU",
      "value" : "<VMware IT department>"
    }, {
      "key" : "C",
      "value" : "<US>"
    }, {
      "key" : "ST",
      "value" : "<California>"
    }, {
      "key" : "L",
      "value" : "<Palo Alto>"
    } ]
  },
  "algorithm" : "RSA",
  "key_size" : 3072,
  "pem_encoded" : "-----BEGIN CERTIFICATE REQUEST-----\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n<STRING>\n-----END CERTIFICATE REQUEST-----\n",
  "is_ca" : false,
  "resource_type" : "csr",
  "id"           : "<Certificate ID>",
  "display_name" : "<Certificate ID>",
  "_create_time" : <EPOCH_TIME>,
  "_create_user" : "admin",
  "_last_modified_time" : <EPOCH_TIME>,
  "_last_modified_user" : "admin",
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_revision" : 0
   
   
3. Edit the CSR into a readable file
   
   For traceability I'll create a file referencing the change NSX02_FQDN-n.csr
   The CSR is generated with "\n" line breakers, these have to be removed via a text editor like vi, vim, Notepad++, Sublime  etc.
   CSR should look like this
-----BEGIN CERTIFICATE REQUEST-----
<STING><STRING><STRING><STRING><STRING><STRING><STRING><STING><STRING><STRING><STRING><STRING><STRING><STRING><STING><STRING><STRING><STRING><STRING><STRING><STRING><STING><STRING><STRING><STRING><STRING><STRING><STRING><STING><STRING><STRING><STRING><STRING><STRING><STRING><STING><STRING><STRING><STRING><STRING><STRING><STRING><STING><STRING><STRING><STRING><STRING><STRING><STRING>
-----END CERTIFICATE REQUEST----- 
   
   
4. Verify the FQDN, SAN names and the public key
   

   root@NSX02_FQDN:~# openssl req -in NSX02_FQDN-n.csr -noout -text | grep -i -A1 "X509v3 Subject Alternative Name:"
            X509v3 Subject Alternative Name:
                DNS:<NSX_VIP>, DNS:<NSX01_FQDN >, DNS:<NSX02_FQDN>, DNS:<NSX03_FQDN>


   - We can also check the public key, as we nee to compare it with the Signed certificate
root@NSX02_FQDN:~# openssl req -in NSX02_FQDN-n.csr -pubkey -noout -outform pem | sha256sum
<alpha_numeric_string>  -

   
   
   
5. Get Certificate signed by the CA based on the CSR
   Customer action, no input from VMware
   
   
6. Compare private Key - should match with above CSR
   
   root@NSX02_FQDN:~# openssl x509 -in NSX02_FQDN.crt -pubkey -noout -outform pem | sha256sum
<alpha_numeric_string>  -   <= should match  value from Step 4
   
   
7. Import the Certificate to the NSX-T UI
   Follow the path 
   NSX02_FQDN_UI > Systems  > Certificates > CSRs
   Use the "id" from NSX02_FQDN.csr to identify the correct CRS
   Select Import Certificate for CSR - Browse for the .crt file or copy paste the string 
   
   [!!!]N.B. it is recommended to use the complete certificate chain, as shown below[!!!]
   -----BEGIN CERTIFICATE-----
Leaf STRING                                                      <-----Certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Intermediate STRING                                              <-----Intermediate Certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Root STRING                                                      <-----Root Certificate
-----END CERTIFICATE-----
   Turn off Service certificate
   
   
   
   
8. Obtain certificate ID from NSX UI
   
   NSX02_FQDN_UI > Systems  > Certificates > Certificates 
   Use the same "id" from NSX02_FQDN.csr to identify the correct Certificate 
   Select the 3 dots, open the menu and copy the Certificate ID
   You can see/compare the expiration date - should match your Obtained certificate, also you can even check the certificate chain.
   
   
9. POST  the certificate via API call
   
   SSH on the NSX02_FQDN Node, run CLI 
   curl -v \
     -k \
     -X POST \
     "https://<NSX02_FQDN>/api/v1/node/services/http?action=apply_certificate&certificate_id=<Certificate ID from Step 8>" \
     -u 'admin:<admin_password>' \
     -H 'Content-Type: application/xml'
   
   
10. Certificate should be in "used" Status
    
    refresh site and verify the certificate in the Web GUI 
    NSX02_FQDN_UI > Systems  > Certificates > Certificates -  use the same "id" from NSX02_FQDN.csr - "Where used" will change to 1
    
    
    
11. Repeat the steps from 1 to 10 for the rest of the Nodes
    There is no particular order and it's not relevant where the VIP is running
    
    
    
12. NSX_VIP
    Again there is no particular order, VIP can be first, last, or in the middle. 
    For continuity purposes, I've replaced the nodes first, in no particular order, and I left the NSX VIP last as this process has 1 difference
    Follow Steps.1 through Step.8 for NSX_VIP
    
    
13. Post the certificate via API call
    Notice the slight difference between the VIP and the Node.
    
    curl -v \
     -k \
     -X POST \
     "https://<NTX_VIP>/api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=<Certificate ID from UI>" \
     -u 'admin' \
     -H 'Content-Type: application/xml'
    
    
    
    

Additional Information

An additional workaround is to replace the certificate from the NSXT UI

The problem remains, that the GUI replacement doesn't handle multiple SANs  
https://techdocs.broadcom.com/us/en/vmware-cis/nsx/nsxt-dc/3-2/administration-guide/certificates/importing-certificates/replace-certificates-through-api.html

Possible additional step is to update the VCF trust store
Said step might not be necessary, as in most cases, as the same CA should also be signing other SDDC component certificates including the SDDC.
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/4-5/administering/certificate-management-admin/install-third-party-ca-signed-certificates-in-vmware-cloud-foundation-using-the-legacy-method-admin.html
Simply put if we already TRUST the CA that is signing the NSX certificates no further action is required