ingress certificate failed to delete on NSX-T Manager due to "error_message" : "One or more relationships exist for object with id Certificate.

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware NSX-T Data Center

Issue/Introduction

• HTTP ingress certificate is about to expire.
• When a new secret and ingress certificate are created, this old certificate is used instead of the new one.
• You are unable to delete the stale ingress certificate on NSX-T.
• You are making a DELETE API call to delete the certificate, e.g.:
  DELETE /api/v1/trust-management/certificates/<certificate_UUID> 
or
DELETE /policy/api/v1/trust-management/certificates/<certificate_UUID>
• The API will fail with error:
  "httpStatus": "BAD_REQUEST", 
"error_code": 3022, 
"module_name": "internal-framework", 
"error_message": "One or more relationships exist for object with id Certificate/<certificate_UUID>."

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

• In NCP version earlier than 4.0.1, there is insufficient pagination for certificates in NCP, where all certificate data are not loaded due to NCP was paginating data.
• There is an old certificate being referenced in the virtual server, which is preventing deletion of the old certificate.

Resolution

This pagination issue is fixed from NCP 4.0.1.

The latest TKGi 1.15.4 is shipping NCP 4.0.1.2 as per https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid-integrated-edition/1-20/tkgi/release-notes.html#product-snapshot-1

Workaround:

You need to remove any old certificate reference from the virtual server, to allow the old certificate to be deleted.

1. bosh ssh into the master VM from the TKGi cluster
   
   

   bosh -d service-instance_<GUID> ssh master/0
   sudo -i

   
   
   
2.  load the variables required to send NSX api calls
   
   

   source /var/vcap/jobs/pks-nsx-t-prepare-master-vm/bin/pre-start

   
   
   
3. backup the old certificate <old_certificate_id>
   
   

   curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/trust-management/certificates/<certificate_id>"  --cert ${NSX_MANAGER_CLIENT_CERT_FILE}  --key ${NSX_MANAGER_CLIENT_KEY_FILE}  --cacert ${NSX_MANAGER_CA_CERT_FILE}  -H "accept: application/json"  -H "Content-Type: application/json" > backup_certificate.json

   
   
   
   
4.  list all virtual-server
   
   

   curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/loadbalancer/virtual-servers/" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE}

   
   
   
5.  Backup the virtual server <virtual_server_id> that is still referencing the old certificate
   
   

   curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/loadbalancer/virtual-servers/<virtual_server_id>" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} > backup_virtual-server.json
   cp backup_virtual-server.json patch_virtual-server.json

6. Delete the old certificate id in the file in section "client_ssl_profile_binding" in the file new_virtual-server.json then patch the virtual server
   
   

   curl -X PUT "https://${NSX_MANAGER_HOST}/api/v1/loadbalancer/virtual-servers/<virtual_server_id>" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} -H "content-type:application/json" -d @patch_virtual-server.json

   
   
   
7. Delete the old certificate
   
   

   curl -X DELETE "https://${NSX_MANAGER_HOST}/api/v1/trust-management/certificates/<old_certificate_id>" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --ke ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} -H "X-Allow-Overwrite: true"