Principal Identity certificate with Private Key is expired/about to expire

Products

VMware NSX

Issue/Introduction

This article provide instruction on how to replace/renew the Principle identity certificate which is having the private key attached with them.
Similar to the below output will be visible when running the https://<OPS_MANANGER>/api/v0/deployed/certificates?expires_within=3m:

{
    "configurable": true,
    "is_ca": false,
    "property_reference": ".properties.network_selector.nsx.nsx-t-superuser-certificate",
    "property_type": "rsa_cert_credentials",
    "product_guid": "pivotal-container-service-########",
    "location": "ops_manager",
    "variable_path": null,
    "issuer": "/CN=########",
    "valid_from": "####-##-######",
    "valid_until": "####-##-######"
}

Environment

VMware NSX-T Data Center

Cause

The customer manually provisioned a certificate on the Tanzu appliance. However, due to a discrepancy between the certificate metadata and the expected values, NSX began triggering certificate expiration alerts.

Resolution

The certificate is located on the TKGI TILE > Networking > NSX > NSX Manager Super User Principal Identity Certificate.
You need two things to renew the certificate:
- The certificate_id of the expired superuser certificate.
- The ID of the user (principal identity) whose certificate needs to be renewed.

1- Get the Certificate ID of the expired certificate from the NSX-UI by navigating to System > Certificate

2- Get the ID of the user (Principal Identity ID) whose certificate needs to be replaced by running the below command:-

curl -X GET -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/principal-identities | jq -r '.results[]| select(.certificate_id == "<Certificate-ID>")'

3- Create the new nsx-t-superuser-certificate for the same (old) User Principal ID

Note: Run the create_certificate.sh script that will create the certificate and private key and upload the certificate to NSX Manager. After running this script, you will have the certificate file and private key in the local path and the certificate will be posted to the NSX Manager.

Create a directory called nsxt-pi-cert
```
mkdir ~/nsxt-pi-cert
```
Navigate to the directory
```
cd ~/nsxt-pi-cert
```
Download the create_certificate.sh.zip file attached to this article and extract the contents (create_certificate.sh). Copy this script to the system where you will run the commands from.
Open the script in a text editor and make the following changes:
1. Set NSX_MANAGER to the NSX-T manager IP address.
2. Set NSX_USER to the "admin" account.
3. The original PI_NAME is "pks-nsx-t-superuser" according to initial installation document, please set PI_NAME like "pks-nsx-t-superuser_##" to make a difference, ## could be like 01, 02, etc.
4. Set NSX_SUPERUSER_CERT_FILE to the name of the new certificate file.
5. Set NSX_SUPERUSER_KEY_FILE to the name of the new private key file.
6. Update the "CN=" reference, updating the value to match thew update PI_NAME value.

4- Make the sh script excludable

chmod +x create_certificate.sh

5- Run the sh script

./create_certificate.sh

Note: The new certificate will get uploaded to the NSX-T with the Issuer Name "pks-nsx-t-superuser_##". In this example, the issuer is "pks-nsx-t-superuser_01". The script will create two files "pks-nsx-t-superuser_01.crt" and "pks-nsx-t-superuser_01.key". Make a copy of these files as they will be used to update the TKGi/PKS Tile with the new NSX-T Superuser certificate in step 4.

6- Get the new NSX-t-superuser-certificate "id" value by running the below command:-

curl -X GET -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/certificates | jq -r '.results[] | select(.display_name == "<PI_NAME>")'

7- Bind the new certificate ID to the principal identity ID

Create a bind.json file containing the principal identity ID obtained in Step 2 and new certificate ID obtained in Step 6.

{
"principal_identity_id": "######",
"certificate_id": "######"
}

Run the below command to bind the certificate to the principal id:

curl -X POST -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/principal-identities?action=update_certificate -H "Content-Type: application/json" -H "X-Allow-Overwrite: true" -d @bind.json

Run the below command to validate that the tkgi-nsx-t-superuser principal identity identity ID is bound to the new certificate.

curl -k -X GET -u 'admin:<Password>' -k https://<NSX_MANAGER>/api/v1/trust-management/principal-identities | jq -r '.results[]| select(.id == "<PRINCIAL_IDENTITY_ID>")'

8- Update the TKGi/PKS Tile with the new NSX-T Superuser certificate.

Login to the Ops Manager UI click on the TKGI/PKS tile > Networking.
Find the "NSX Manager Super User Principal Identity Certificate" section.
Click on Change to update the NSX-T superuser certificate and Private Key.
Update the "Certificate" filed with the content in the "pks-nsx-t-superuser_##.crt" file and the "Private Key" filed with the content in the "pks-nsx-t-superuser_##.key" file. (Both files should be located in the location where you ran the " create_certificate.sh" script)
Save

9- Click on Installation Dashboard and select Review Pending Changes.
1- On the Review Pending Changes page, expand ERRANDS under "Tanzu Kubernetes Grid Integrated Edition" (or "Enterprise PKS" In version 1.7 and earlier) and enable the "Upgrade all clusters errand" > Click on Apply Changes.

Additional Information

If you see the below logs/error while running the script, make sure you give the correct hostname/IP while Creating the new NSX-t-superuser-certificate for the same (old) User Principal ID (Step 3)

####.##.#### Error building client catalog: [GET /node][500] readNodePropertiesInternalServerError  &{RelatedAPIError:{Details:Client certificate not found in trust store ErrorCode:99 ErrorData:<nil> ErrorMessage:Internal server error has occurred. ModuleName:common-services} RelatedErrors:[]}time="####.####.####>" level=error msg="Failed to get node properties" pks-networking=networkManager

On script editor you may see the error output as below which also indicates that the given hostname is not correct.