After removing a guest cluster some cert-manager controlled objects are left behind.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

A Guest cluster was removed but showing as deleting. After clean up, there were  still orphaned cert-manager  controlled objects which keep recreating. In this case the orphaned cert-manager  controlled objects were Certificate and a CertificateRequest 

Environment

vCenter v8.0.2

vSphere with Tanzu

Cause

cert-manager  controlled objects have annotations which alerts cert-manager  to recreate them when deleted.

See the example annotations for a CertificateRequest managed by cert-manager  

annotations:

    cert-manager.io/certificate-name:  <cluster-name>-metrics-server-cert
    cert-manager.io/certificate-revision: "4"
    cert-manager.io/private-key-secret-name: <cluster-name>-metrics-server-cert-xxxx

The cert-manager webhook stops manually editing/removing of these annotations.

Resolution

Scale down the cert-manager deployments, delete the cert-manager  controlled objects and scale up  the cert-manager deployments

1. Check and list the issuers to be removed exist
kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges -A | grep -E "NAME|<cluster-name>"
kubectl get secrets -n <cluster namespace> | grep -E "NAME|<cluster-name>"

2. Check the cert-manager pods and deployments

kubectl get all -n vmware-system-cert-manager
 
2. Scale down the cert-manager pods

kubectl scale deployment.apps/cert-manager -n vmware-system-cert-manager  --replicas=0
kubectl scale deployment.apps/cert-manager-cainjector  -n vmware-system-cert-manager  --replicas=0
kubectl scale deployment.apps/cert-manager-webhook  -n vmware-system-cert-manager  --replicas=0

 

3. Delete certificates etc *
kubectl delete -n <cluster namespace>  certificate.cert-manager.io/<cluster cert>
kubectl delete -n <cluster namespace>  certificaterequest.cert-manager.io/<cluster cert request>
kubectl delete -n <cluster namespace>  secret <cert-manager private-key-secret-name for cluster> 

 

4. check the number of replicas and scale back   up 
kubectl scale deployment.apps/cert-manager -n vmware-system-cert-manager  --replicas=1
kubectl scale deployment.apps/cert-manager-cainjector  -n vmware-system-cert-manager  --replicas=1
kubectl scale deployment.apps/cert-manager-webhook -n vmware-system-cert-manager  --replicas=1

5. Check the certs etc have been removed
kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges -A | grep -E "NAME|<cluster-name>"