Renewing k8s-msg-client self-signed certificate on NSX Manager UI

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

NSX Intelligence common agent(running within the NSX Manager appliance) uses a self signed Kafka messaging client certificate with name 'k8s-msg-client'. During NSX upgrade prechecks, we might encounter the following error notifying the expiry of k8s-msg-client self certificate on NSX.

 

Environment

NSX versions <= 4.1.x versions

NAPP 4.x

Cause

NSX does not replace the k8s-msg-client certificate autiomatically since the node ID used by this certificate does not belong to any of the NSX Manager node IDs

Resolution

The following procedure outlines the manual steps to renew the k8s-msg-client certificate.

NOTE : This procedure is applicable to NSX versions <= 4.1.x versions as NSX versions >= 4.2.x have an inbuilt option in the UI itself to replace the expired certificate as outlined in :

https://knowledge.broadcom.com/external/article/375300

 

Procedure

1. Backup the current expired k8s-msg-client certificate using the export option from NSX UI

2. Make note of following certificate parameters of the existing k8s-msg-client certificate :

Certificate Name in NSX UI
Common Name (same as Issued By)
Organization Name 
Organization Unit 
Locality 
State
Algorithm
Key Size

3. From the NSX UI create a new certificate with the same parameters noted above and make sure to uncheck the box for "Service Certificate"

 

4. Now , Get the node ID of the expired certificate as we need to associate the new certificate with the same node ID.

On the Certificates page, under Used By column, click on '1' hyperlink against the expired certificate and copy the node ID 

As an alternate, the node_id andcert-UUID  can be fetched with the GET API call - https://<nsx-mgr-ip>/api/v1/trust-management/certificates/<Certificate-ID>

5. From NSX UI, get the certificate ID  of the new certificate generated in step 3 

 

6.Using Postman API tool, run the following POST API call to replace the expired certificate with Node ID and Certificate ID captured in step 4 & 5

<nsx-mgr-ip>/api/v1/trust-management/certificates/<Certificate-ID>?action=apply_certificate&service_type=K8S_MSG_CLIENT&node_id=<Node-ID>

 

7. Verify that the status of the POST API call is 200 OK. 

8. Once the certificate is replaced successfully, NAPP services(specifically kafka and its dependent services) will be restarted

9. Optionally delete the expired certificate, ensure the Used By column against the expired certificate is '0' and the same is '1' on the new certificate before deleting the certificates

 

 

Additional Information

In the case of Tanzu/WCP and NAPP have been cleaned up and  "K8S_MSG_CLIENT"  expired alarm  is still showing in NSX manager then you can refer below KB >  
https://knowledge.broadcom.com/external/article/393976/renewing-k8smsgclient-selfsigned-certifi.html