In a Kubernetes environment, Ingress Certificates can be stuck in "DELETION IN PROGRESS"

Products

• VMware NSX

Issue/Introduction

• There is a Kubernetes solution integrated with NSX-T.
• Ingress are used in the Kubernetes environment with Secrets and they are imported into NSX-T.
• The certificates of deleted ingress remained in the NSX-T web UI but are greyed-out.
• There is an icon which indicates that the certificates are in  a "DELETION IN PROGRESS" state.
• In the NSX-T Manager log: /var/log/policy/policy.log, the following messages can be seen:
  
  

  2021-05-28T19:10:13.551Z  INFO providerTaskExecutor-40 AlarmServiceImpl - POLICY [nsx@#### comp="nsx-manager" level="INFO" subcomp="policy"] Message returned Realization failure, waiting for realization of [{TlsCertificate}]: path=[/infra/certificates/lb_########-####-####-####-############_ntmme], Realization will be reattempted in next cycle (max 5 minutes)

  
  *The path indicates the certificate(s) affected by this behavior.
  ** If this message is seen once and after 5 minutes, the objects is deleted. You can ignore it.

Environment

VMware NSX-T Data Center
VMware NSX Container Plugin (NCP)

Cause

This issue is likely to occur when: 

•  The same ingress certificates are used or re-used by the ingress.
•  Ingress is deleted and within the same timeframe (milliseconds), the ingress is re-created with the same certificate.

Inside NSX, the certificate is still in deletion. Before the deletion is complete, it gets attached to a load nalancer hence, the deletion cannot be done.

Resolution

This issue is resolved in NCP 3.2

To workaround this issue, the certificate will have to be deleted via the NSX-T REST API.

1. Consult list of certificates mark for deletion: 
   
   GET https://{{ip-address}}/policy/api/v1/infra/certificates/?include_mark_for_delete_objects=true

From the results, identify the UUID of the stale certificate. 
   
   
2. If the certificate is not in use anymore, use the following API to delete the certificate: 
   
   DELETE https://{{ip-address}}/policy/api/v1/infra/certificates/lb_########-####-####-####-############_hvwpn