NSX Load Balancer Certificate Rotation for WCP/vSphere with Tanzu using NSX-T

Products

• VMware vSphere Kubernetes Service
• VMware NSX

Issue/Introduction

• The validity period of the NSX Load Balancer certificate has expired in vCenter vSphere Client under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer.
• The validity period of the lb-domain-#### certificate issued to defaultingress.local has expired in NSX Manager under System -> Certificates
• The validity period  of lb-default.cert certificate in any of the supervisor control plane VM under /etc/vmware/wcp/tls/ncp/ is expired.

Resolution

Create an L7 ingress resource to trigger the creation of an L7 Virtual Server. When you rotate the certificate using the following steps, NCP replaces the existing/old certificate.

Disclaimer:

1. If no L7 ingress resources are in use by the Supervisor or TKG clusters, and NSX is not utilized for L7 services, skip the Create Ingress section and proceed directly to REPLACE CERT.
2. If an Ingress resource already exists, skip the following steps and go to the REPLACE CERT section

Create Ingress:

1. Create an L7 ingress resource as follows:
   

     apiVersion: networking.k8s.io/v1
     kind: Ingress
     metadata:
       name: test-ingress
       annotations:
         kubernetes.io/ingress.class: "nsx"
     spec:
       tls:
       - hosts:
         - test.example.com
         secretName: test-secret
       rules:
       - host: test.example.com
         http:
           paths:
           - path: /tmp
             pathType: ImplementationSpecific
             backend:
               service:
                 name: test-svc
                 port:
                   number: 80 

2. Run the following command once the file ingress.yaml is populated with the preceding content: kubectl apply -f ingress.yaml

3. Confirm ingress creation: kubectl get ingress -A

4. Confirm that the L7 HTTP Virtual Server exists on NSX Manager. Navigate to Networking > Load Balancing > Virtual Servers and filter for Type: L7 HTTP.

Replace/Rotate the certificate: 

1. Generate CSR from vCenter vSphere Client under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer Pane -> Actions > Generate CSR

2. Provide the details for the certificate. Once the CSR is generated, click Copy.

3. Copy CSR to the vCenter in order to get it signed by VMCA. (you can get the CSR signed by your own CA if you are using custom CA)

4. Login to the vCenter through SSH using the root

5. Type shell and press Enter.

6. Generate the certificate using the above CSR: /usr/lib/vmware-vmca/bin/certool --gencertfromcsr --csrfile cert.csr --cert lb-renew.crt

7. Copy the content of lb-renew.crt and replace it under Actions > Replace Certificate.

8. Check NSX Manager -> System -> Certificates. The new certificate should be visible.

   Note: If you have not created Ingress initially or not followed Create Ingress steps and directly tried to REPLACE CERT, then there would be two certificates on NSX manager. Follow the steps in "Additional information" to delete the old certificate

9. Confirm the new certificate is imported. NSX Manager automatically removes the old certificate under System > Certificates.

10. Delete the ingress created in Step 1 with command: kubectl delete -f ingress.yaml

Note: This does not require an NCP restart.

Additional Information

Note: You cannot access the L7 ingress resources deployed across the clusters (Supervisor and TKGs) during this process because NSX uses this certificate as the default certificate for HTTPS traffic for L7 Ingress.

Removal of Old Certificate from NSX

To delete the old certificate if it is still present (as per Step 6):

1. In NSX manager, copy the old certificate ID with the name lb-domain-#### issued to defaultingress.local under System > Certificates. Alternatively, fetch this using the following command: GET /policy/api/v1/infra/certificates/

2. Compare pem_encoded or _create_time in the returned JSON.

3. Use the following API call to remove the expiring certificate: curl -k -u 'admin:<NSX_admin_Password>' -X DELETE https://<NSX_Manager>/policy/api/v1/infra/certificates/<Certificate_ID> -H "X-Allow-Overwrite:true"

4. NSX performs a dependency check and blocks deletion if the certificate is in use.

5. Restart the NCP pod. Restart the NCP pod by either touching the NCP deployment or deleting the NCP pod. This is required to sync the NCP certificate cache and prevents the stale certificate from being referenced in future L7 VirtualServer creation calls.