NSX Load Balancer Certificate Rotation for WCP/vSphere with Tanzu using NSX-T
Article ID: 326382
Updated On:
Products
Issue/Introduction
Symptoms:
- You would notice the validity of the NSX Load Balancer certificate has expired in vCenter UI under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer.
- You would notice the validity of the lb-domain-XXXX certificate issued to defaultingress.local has expired in NSX Manager under System -> Certificates
- You would notice the validity of lb-default.cert certificate in any of the supervisor control plane VM under /etc/vmware/wcp/tls/ncp/ is expired.
Environment
Resolution
- If you have an Ingress resource that exists already, you can skip the below steps and go to REPLACE CERT section.
Create Ingress:
1. Create L7 ingress resource as below:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" spec: tls: - hosts: - cafe.example.com secretName: cafe-secret rules: - host: cafe.example.com http: paths: - path: /tea pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80
Run the below command once the file ingress.yaml is populated with the above content
- kubectl apply -f ingress.yaml
Confirm ingress creation using the below command
- kubectl get ingress -A
2. Confirm that L7 HTTP Virtual Server exists on NSX-T Manager. Go to NSX Manager Networking -> Load Balancing -> Virtual Servers filter for Type L7 HTTP
3. Rotate the certificate using the REPLACE CERT section below
4. Confirm new cert imported and old cert should be automatically removed in NSX Manager under System -> Certificates
5. Delete ingress created in step 1
- kubectl delete -f ingress.yaml
Note: This should not require any restart in NCP
REPLACE CERT:
1. Generate CSR from vCenter UI under Cluster -> Configure -> Supervisor Cluster -> Certificates -> NSX Load Balancer Pane -> Actions > Generate CSR
2. Provide the details for the certificate. Once the CSR is generated, click Copy.
3. Copy CSR to vCenter in order to get it signed by VMCA. (you can get the CSR signed by your own CA if you are using custom CA)
4. Execute /usr/lib/vmware-vmca/bin/certool --gencertfromcsr --csrfile cert.csr --cert lb-renew.crt
5. Copy the content of lb-renew.crt and replace it under Actions > Replace Certificate.
6. Check NSX Manager -> System -> Certificates. New certificate should appear.
Note:- If you have not created Ingress initially or not followed Create Ingress steps and directly tried to REPLACE CERT, then there would be two certificates on NSX manager. Follow the below workaround to delete the old cert.
Workaround:
Here's a workaround to delete old cert:
1. In NSX manager copy the old cert Id with name lb-domain-XXXX issued to defaultingress.local under System -> Certificates. Alternatively, this can be fetched by "GET /policy/api/v1/infra/certificates/" API call and compare pem_encoded or _create_time in the returned JSON.
2. Use the below API call to remove the expiring cert. NSX will do a dependency check and block deletion if the cert is used anywhere.
curl -k -u 'admin:<NSX_admin_Password>' -X DELETE https://<NSX_Manager>/policy/api/v1/infra/certificates/<Certificate_ID> -H "X-Allow-Overwrite:true"
Example:-
curl -k -u 'admin:VMware1!VMware1!' -X DELETE https://nsxmgr.corp.local/policy/api/v1/infra/certificates/lb_domain-c8:07e2e4d5-2c90-4591-98e3-6f656a67e3fc_ahtwa -H "X-Allow-Overwrite:true"
3. You have to restart the NCP pod. NCP can be restarted by either touching NCP deployment or deleting NCP pod. This is required to sync the NCP certificate cache and will prevent the stale certificate from being referenced in future L7 VirtualServer creation calls.
Additional Information
You won't be able to access the L7 ingress resources deployed across the clusters(Supervisor and TKGs) because in NSX-T this certificate is used only for L7 Ingress as the default certificate for HTTPS traffic.
Feedback
