NSX Load Balancer Certificate Rotation for WCP/vSphere with Tanzu using NSX-T

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service
• VMware NSX

Issue/Introduction

• The validity period of the NSX Load Balancer certificate has expired in vCenter UI under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer.
• The validity period of the lb-domain-XXXX certificate issued to defaultingress.local has expired in NSX Manager under System -> Certificates
• The validity period  of lb-default.cert certificate in any of the supervisor control plane VM under /etc/vmware/wcp/tls/ncp/ is expired.

Environment

• Supervisor 7
• Supervisor 8
• Supervisor 9
• VMware NSX

Resolution

Create an L7 ingress resource which will trigger the creation of L7 Virtual Server and when the certificate is rotated using the following steps, NCP will replace the existing/old certificate.

Disclaimer:

If no L7 ingress resources are being used by the Supervisor or TKG clusters, and NSX-T is not being utilized for L7 services, you may skip the "Create Ingress" step and proceed directly to the "REPLACE CERT" section.

If you have an Ingress resource that exists already, you can skip the following steps and go to REPLACE CERT section.

Create Ingress:

1. Create L7 ingress resource as below:
   

     apiVersion: networking.k8s.io/v1
     kind: Ingress
     metadata:
       name: cafe-ingress
       annotations:
         kubernetes.io/ingress.class: "nsx"
     spec:
       tls:
       - hosts:
         - cafe.example.com
         secretName: cafe-secret
       rules:
       - host: cafe.example.com
         http:
           paths:
           - path: /tea
             pathType: ImplementationSpecific
             backend:
               service:
                 name: tea-svc
                 port:
                   number: 80 

2. Run the following command once the file ingress.yaml is populated with the preceding content
   
   kubectl apply -f ingress.yaml
   
   
3. Confirm ingress creation using the following command:
   
   kubectl get ingress -A
   
   
4. Confirm that L7 HTTP Virtual Server exists on NSX-T Manager. Go to NSX Manager Networking -> Load Balancing -> Virtual Servers filter for Type L7 HTTP


5. Rotate the certificate using the REPLACE CERT section below
   
   
6. Confirm new certificate is imported and old certificate should be automatically removed in NSX Manager under System -> Certificates


7. Delete ingress created in step 1
   
   kubectl delete -f ingress.yaml

Note: This should not require any restart in NCP

REPLACE CERT:

1. Generate CSR from vCenter UI under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer Pane -> Actions > Generate CSR



2. Provide the details for the certificate. Once the CSR is generated, click Copy.
   
   
3. Copy CSR to the vCenter appliance in order to get it signed by VMCA. (you can get the CSR signed by your own CA if you are using custom CA)
   
   
4. Execute: /usr/lib/vmware-vmca/bin/certool --gencertfromcsr --csrfile cert.csr --cert lb-renew.crt
   
   
5. Copy the content of lb-renew.crt and replace it under Actions > Replace Certificate.
   
   
6. Check NSX Manager -> System -> Certificates. The new certificate should be visible.
   
   Note: If you have not created Ingress initially or not followed Create Ingress steps and directly tried to REPLACE CERT, then there would be two certificates on NSX manager. Follow the steps in "Additional information" to delete the old certificate

Additional Information

Note:

You won't be able to access the L7 ingress resources deployed across the clusters(Supervisor and TKGs) because in NSX-T this certificate is used only for L7 Ingress as the default certificate for HTTPS traffic.

Removal of old certificate from NSX:

(to delete the old certificate if it is sill present, per the Note in Step 6 of the REPLACE CERT section)

1. In NSX manager copy the old certificate Id with name lb-domain-#### issued to defaultingress.local under System -> Certificates. Alternatively, this can be fetched using the following command:
   
   GET /policy/api/v1/infra/certificates/

compare pem_encoded or _create_time in the returned JSON.
   
   
2. Use the following API call to remove the expiring certificate:
   
   curl -k -u 'admin:<NSX_admin_Password>' -X DELETE https://<NSX_Manager>/policy/api/v1/infra/certificates/<Certificate_ID> -H "X-Allow-Overwrite:true

   NSX will do a dependency check and block deletion if the certificate is used anywhere.
   
   
3. Restart the NCP pod. The NCP pod can be restarted by either touching NCP deployment or deleting NCP pod. This is required to sync the NCP certificate cache and will prevent the stale certificate from being referenced in future L7 VirtualServer creation calls.