NSX Load Balancer Certificate Rotation for WCP/vSphere with Tanzu using NSX-T

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu
• VMware NSX

Issue/Introduction

• The validity period of the NSX Load Balancer certificate has expired in vCenter UI under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer.
• The validity period of the lb-domain-XXXX certificate issued to defaultingress.local has expired in NSX Manager under System -> Certificates
• The validity period  of lb-default.cert certificate in any of the supervisor control plane VM under /etc/vmware/wcp/tls/ncp/ is expired.

Environment

VMware vSphere 7.0 with Tanzu
VMware NSX

Resolution

Create an L7 ingress resource which will trigger the creation of L7 Virtual Server and when the certificate is rotated using the following steps, NCP will replace the existing/old certificate.
If you have an Ingress resource that exists already, you can skip the following steps and go to REPLACE CERT section.

Create Ingress:

1. Create L7 ingress resource as below:
   
   

     apiVersion: networking.k8s.io/v1
     kind: Ingress
     metadata:
       name: cafe-ingress
       annotations:
         kubernetes.io/ingress.class: "nsx"
     spec:
       tls:
       - hosts:
         - cafe.example.com
         secretName: cafe-secret
       rules:
       - host: cafe.example.com
         http:
           paths:
           - path: /tea
             pathType: ImplementationSpecific
             backend:
               service:
                 name: tea-svc
                 port:
                   number: 80 

   
   Run the following command once the file ingress.yaml is populated with the preceding content
   
   kubectl apply -f ingress.yaml
   
   Confirm ingress creation using the following command
   
   kubectl get ingress -A
   

2. Confirm that L7 HTTP Virtual Server exists on NSX-T Manager. Go to NSX Manager Networking -> Load Balancing -> Virtual Servers filter for Type L7 HTTP
3. Rotate the certificate using the REPLACE CERT section below
4. Confirm new certificate is imported and old certificate should be automatically removed in NSX Manager under System -> Certificates
5. Delete ingress created in step 1
   
   kubectl delete -f ingress.yaml
   
   Note: This should not require any restart in NCP

REPLACE CERT:

1. Generate CSR from vCenter UI under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer Pane -> Actions > Generate CSR
2. Provide the details for the certificate. Once the CSR is generated, click Copy.
3. Copy CSR to the vCenter appliance in order to get it signed by VMCA. (you can get the CSR signed by your own CA if you are using custom CA)
4. Execute /usr/lib/vmware-vmca/bin/certool --gencertfromcsr --csrfile cert.csr --cert lb-renew.crt
5. Copy the content of lb-renew.crt and replace it under Actions > Replace Certificate.
6. Check NSX Manager -> System -> Certificates. The new certificate should be visibile.
   
   Note:- If you have not created Ingress initially or not followed Create Ingress steps and directly tried to REPLACE CERT, then there would be two certificates on NSX manager. Follow the below workaround to delete the old cert.

Workaround:

(to delete the old certificate if it is sill present, per the Note in Step 6 of the REPLACE CERT section)

1. In NSX manager copy the old certificate Id with name lb-domain-#### issued to defaultingress.local under System -> Certificates. Alternatively, this can be fetched by "GET /policy/api/v1/infra/certificates/" API call and compare pem_encoded or _create_time in the returned JSON.
2. Use the following API call to remove the expiring certificate. NSX will do a dependency check and block deletion if the cert is used anywhere.
   
   curl -k -u 'admin:<NSX_admin_Password>' -X DELETE https://<NSX_Manager>/policy/api/v1/infra/certificates/<Certificate_ID> -H "X-Allow-Overwrite:true"
   
3. Restart the NCP pod. NCP can be restarted by either touching NCP deployment or deleting NCP pod. This is required to sync the NCP certificate cache and will prevent the stale certificate from being referenced in future L7 VirtualServer creation calls.
   
   

Additional Information

Impact/Risks:
You won't be able to access the L7 ingress resources deployed across the clusters(Supervisor and TKGs) because in NSX-T this certificate is used only for L7 Ingress as the default certificate for HTTPS traffic.