How to renew the nsx-t-superuser-certificate used by Principal Identity user

Products

VMware Tanzu Kubernetes Grid Integrated Edition VMware NSX

Issue/Introduction

This article provides instructions for renewing the certificate used by NSX-T principal identity user created for Tanzu Kubernetes Grid Integrated Edition.

Target certificates

.properties.network_selector.nsx.nsx-t-superuser-certificate
.iaas_configuration.nsx_t_client_certificate (Follow step.10 in Resolution section)

Symptoms:

You see information similar to the following when navigating to https://<OPS_MANANGER>/api/v0/deployed/certificates?expires_within=3m :

Output:

...
{
    "configurable": true,
    "is_ca": false,
    "property_reference": ".properties.network_selector.nsx.nsx-t-superuser-certificate",
    "property_type": "rsa_cert_credentials",
    "product_guid": "pivotal-container-service-########",
    "location": "ops_manager",
    "variable_path": null,
    "issuer": "/CN=########",
    "valid_from": "2020-07-02T13:41:10Z",
    "valid_until": "2021-07-02T13:41:10Z"
}
...

Environment

VMware Tanzu Kubenetes Grid Integrated Edition
VMware NSX-T Data Center

Resolution

Notes:

The certificate is located on the TKGI TILE > Networking > NSX > NSX Manager Super User Principal Identity Certificate.
You need two things to renew the certificate:
- The certificate_id of the expired superuser certificate
- The ID of the user (principal identity ID) whose certificate needs to be renewed

Instructions:

Get the certificate ID:

Note: Make a note of the "Issuer", "valid_from", and "valid_until" values,

Expired data

{
    "configurable": true,
    "is_ca": false,
    "property_reference": ".properties.network_selector.nsx.nsx-t-superuser-certificate",
    "property_type": "rsa_cert_credentials",
    "product_guid": "pivotal-container-service-########",
    "location": "ops_manager",
    "variable_path": null,
    "issuer": "/CN=########",
    "valid_from": "2020-07-02T13:41:10Z",
    "valid_until": "2021-07-02T13:41:10Z"
}

Login to the NSX-T Manager UI and navigate to System > Certificate then find the certificate ID by the issuer [In this case it is "tkgi-nsx-t-superuser"] and confirm the start and expiration date

Note: Make a note of the certificate ID you obtained from the previous step as it will be used in the next step. [Ex: cbfd03e6-44fc-42df-bc71-b771########]

Get The ID of the user (principal ID) whose certificate needs to be replaced by issuing a command similar to the following:

curl -X GET -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/principal-identities | jq -r '.results[]| select(.certificate_id == "<Certificate-ID>")'

You will see output similar to the following:

{

  "name": "########",
  "node_id": "####",
  "permission_group": "undefined",
  "role": "enterprise_admin",
  "certificate_id": "cbfd03e6-44fc-42df-bc71-b771########",
  "is_protected": true,
  "resource_type": "PrincipalIdentity",
  "id": "89819c8b-de0e-4c52-8fa4-dc3b########",
  "display_name": "########",
  "tags": [],
  "_create_user": "admin",
  "_create_time": 1593697486134,
  "_last_modified_user": "admin",
  "_last_modified_time": 1593697486134,
  "_system_owned": false,
  "_protection": "NOT_PROTECTED",
  "_revision": 0

Note: Make a note of the principal identity id - In this example, it is: 89819c8b-de0e-4c52-8fa4-dc3b########

Create the new nsx-t-superuser-certificate for the same (old) User Principal ID

Note: Run the create_certificate.sh script that will create the certificate and private key and upload the certificate to NSX Manager
Note: After running this script, you will have the certificate file and private key in the local path and the certificate will be posted to the NSX Manager.
1. Create a directory called nsxt-pi-cert
```
mkdir ~/nsxt-pi-cert
```
2. Navigate to the directory
```
cd ~/nsxt-pi-cert
```
3. Download the create_certificate.sh.zip file attached to this article and extract the contents (create_certificate.sh). Copy this script to the system where you will run the commands from.
4. Open the script in a text editor and make the following changes:
  1. Set NSX_MANAGER to the NSX-T manager IP address.
  2. Set NSX_USER to the "admin" account.
  3. The original PI_NAME is "pks-nsx-t-superuser" according to initial installation document, please set PI_NAME like "pks-nsx-t-superuser_##" to make a difference, ## could be like 01, 02, etc.
  4. Set NSX_SUPERUSER_CERT_FILE to the name of the new certificate file.
  5. Set NSX_SUPERUSER_KEY_FILE to the name of the new private key file.
  6. Update the "CN=" reference, updating the value to match thew update PI_NAME value.
    
    The following is an example of the changed sections of the file:
```
NSX_MANAGER="<NSX Manager IP>"
NSX_USER="admin"
PI_NAME="pks-nsx-t-superuser_01"
NSX_SUPERUSER_CERT_FILE="pks-nsx-t-superuser_01.crt"
NSX_SUPERUSER_KEY_FILE="pks-nsx-t-superuser_01.key"
-subj /CN=pks-nsx-t-superuser_01
```
5. Make the sh script excludable
  
  chmod +x create_certificate.sh
6. Run the sh script
  ./create_certificate.shNote: Enter the NSX-T admin password when promoted.
  
  You will see output similar to the following:
  
  Password: Generating a 2048-bit RSA private key
  ....+++
  .........................................................................................................+++
  writing the new private key to 'pks-nsx-t-superuser_##.key'
  -----
  {
  "results" : [ {
  "pem_encoded" : "-----BEGIN CERTIFICATE-----\nMII.........laV\n-----END CERTIFICATE-----\n",
  "used_by" : [ ],
  "resource_type" : "certificate_self_signed",
  "id" : "670c81d9-d1e2-4d7f-941a-d08d########",
  "display_name" : "########",
  "tags" : [ ],
  "_create_user" : "admin",
  "_create_time" : 1601567107129,
  "_last_modified_user" : "admin",
  "_last_modified_time" : 1601567107129,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_revision" : 0
  } ]Note: The new certificate will get uploaded to the NSX-T with the Issuer Name "pks-nsx-t-superuser_##". In this example, the issuer is "pks-nsx-t-superuser_01".
  Note: The script will create two files "pks-nsx-t-superuser_01.crt" and "pks-nsx-t-superuser_01.key". Make a copy of these files as they will be used to update the TKGi/PKS Tile with the new NSX-T Superuser certificate in step 5.
7. Get the new nsx-t-superuser-certificate "id" value by running a command similar to the following
  
  curl -X GET -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/certificates | jq -r '.results[] | select(.display_name == "<PI_NAME>")'
  
  In this example, <P_NAME> is `pks-nsx-t-superuser_01`.
  
  You will see output similar to the following:
  
  {
  "pem_encoded": "-----BEGIN CERTIFICATE-----\nMII........laV\n-----END CERTIFICATE-----\n",
  "used_by": [],
  "resource_type": "certificate_self_signed",
  "id": "670c81d9-d1e2-4d7f-941a-d08d########",
  "display_name": "########",
  "tags": [],
  "_create_user": "admin",
  "_create_time": 1601567107129,
  "_last_modified_user": "admin",
  "_last_modified_time": 1601567107129,
  "_system_owned": false,
  "_protection": "NOT_PROTECTED",
  "_revision": 0
  }
  
  Note: Make a note of the nsx-t-superuser-certificate ID. In this example, the value is "670c81d9-d1e2-4d7f-941a-d08d########".
8. Bind the new certificate ID to the principal identity ID
  1. Create a bind.json file containing the principal identity ID obtained in Step 3 and new certificate ID obtained in Step 7.
    
    {
    "principal_identity_id": "######",
    "certificate_id": "######"
    }Using the values in this example, the file would look like the following:
    
    {
    "principal_identity_id": "89819c8b-de0e-4c52-8fa4-dc3b########",
    "certificate_id": "670c81d9-d1e2-4d7f-941a-d08d########"
    }
  2. Run a command similar to the following to bind the certificate to the principal id:
    
    curl -X POST -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/principal-identities?action=update_certificate -H "Content-Type: application/json" -H "X-Allow-Overwrite: true" -d @bind.jsonYou will see output similar to the following:
    
    {
    "name" : "########",
    "node_id" : "#######",
    "permission_group" : "undefined",
    "role" : "enterprise_admin",
    "certificate_id" : "670c81d9-d1e2-4d7f-941a-d08d########",
    "is_protected" : true,
    "resource_type" : "PrincipalIdentity",
    "id" : "89819c8b-de0e-4c52-8fa4-dc3b########",
    "display_name" : "########",
    "tags" : [ ],
    "_create_user" : "admin",
    "_create_time" : 1593697486134,
    "_last_modified_user" : "admin",
    "_last_modified_time" : 1601591521902,
    "_system_owned" : false,
    "_protection" : "NOT_PROTECTED",
    "_revision" : 1
  3. Run a command similar to the following to validate that the tkgi-nsx-t-superuser principal identity identity ID is bound to the new certificate.
    
    curl -k -X GET -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/principal-identities | jq -r '.results[]| select(.id == "<PRINCIAL_IDENTITY_ID>")' In this case <PRINCIAL_IDENTITY_ID> is "89819c8b-de0e-4c52-8fa4-dc3b########".
    You will see output similar to the following:
    
    {
    "name": "########",
    "node_id": "####",
    "permission_group": "undefined",
    "role": "enterprise_admin",
    "certificate_id": "670c81d9-d1e2-4d7f-941a-d08d########", <<<<< The new nsx-t-superuser-certificate From step 7
    "is_protected": true,
    "resource_type": "PrincipalIdentity",
    "id": "89819c8b-de0e-4c52-8fa4-dc3b########",
    "display_name": "########",
    "tags": [],
    "_create_user": "admin",
    "_create_time": 1593697486134,
    "_last_modified_user": "admin",
    "_last_modified_time": 1601591521902,
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 1
    }
9. Update the TKGi/PKS Tile with the new NSX-T Superuser certificate.
  1. Login to the Ops Manager UI click on the TKGI/PKS tile > Networking.
  2. Find the "NSX Manager Super User Principal Identity Certificate" section.
  3. Click on Change to update the NSX-T superuser certificate and Private Key.
  4. Update the "Certificate" filed with the content in the "pks-nsx-t-superuser_##.crt" file and the "Private Key" filed with the content in the "pks-nsx-t-superuser_##.key" file. (Both files should be located in the location where you ran the " create_certificate.sh" script in step 3f)
  5. Save
10. If you are using certificate authentication with NSX-T on the BOSH tile, make sure to update it with the new NSX-T Superuser certificate.
  1. Login to the Ops Manager UI click on the BOSH tile > vCenter Config.
  2. Find the "NSX Manager Principal Identity Certificate" section.
  3. Click on Change to update the NSX-T superuser certificate and Private Key.
  4. Update the "Certificate" filed with the content in the "pks-nsx-t-superuser_##.crt" file and the "Private Key" filed with the content in the "pks-nsx-t-superuser_##.key" file. (Both files should be located in the location where you ran the " create_certificate.sh" script in step 3f)
  5. Save
11. Click on Installation Dashboard and select Review Pending Changes. On the Review Pending Changes page, expand ERRANDS under "Tanzu Kubernetes Grid Integrated Edition" (or "Enterprise PKS" In version 1.7 and earlier) and enable the "Upgrade all clusters errand" > Click on Apply Changes.

Additional Information

You can validate that the new certificate got updated on the master node by running commands similar to the following:

Run the following command on the OpsMan VM to get the Service Deployment UUID

# bosh deployments --column=name | grep service-instance

Run the following command export the nsx-t Superuser cert from the master node.

# bosh -d <service-instance_UUID> scp master/0:/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/########.crt .Ex.

bosh -d service-instance_814f9e24-140d-471e-b4a7-933######## scp master/0:/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/######## .
Run the following command to validate the creation and expiration date of the nsx-t superuser certificate

# openssl x509 -in ########.crt | openssl x509 -issuer -subject -startdate -enddate -nooutNote: You will see output similar to the following:

issuer= /CN=########
subject= /CN=########
notBefore=Oct 2 14:54:21 2020 GMT
notAfter=Oct 2 14:54:21 2022 GMT

Attachments

create_certificate get_app