ingress certificate failed to delete on NSX-T Manager due to "error_message" : "One or more relationships exist for object with id Certificate.

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware NSX-T Data Center

Issue/Introduction

The old certificate that is about to expire, and when a new secret and ingress are created, this old certificate is taken instead of the new one. 

Symptoms:
You can't delete a stale ingress certificate on NSX-T.

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

When invoking the Delete NSX API Call: 

DELETE /api/v1/trust-management/certificates/e3b79442-acf7-4417-a971-2679954bfe4 or /policy/api/v1/trust-management/certificates/e3b79442-acf7-4417-a971-2679954bfe4 NSX API.

You get the below error:
 

DELETE /api/v1/trust-management/certificates/<certificate-id>

    "httpStatus": "BAD_REQUEST",

    "error_code": 3022,

    "module_name": "internal-framework",

    "error_message": "One or more relationships exist for object with id Certificate/e3b79442-acf7-4417-a971-2679954bfe42."

 

This error indicates there is an old certificate reference in the virtual server, that is preventing the old certificate deletion.

Resolution

There is an issue with a lack of pagination for certificate in NCP where all certificate data are not loaded, since NCP was not paginating data.This pagination issue is fixed from NCP 4.0.1
Upgrade to a TKGi version that is using NCP 4.0.1+
The latest TKGi 1.15.4 is shipping NCP 4.0.1.2 as per https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/1.15/tkgi/GUID-release-notes.html#product-snapshot-1

Workaround:
You need to remove any old certificate reference from the virtual server, to allow the old certificate to be deleted.

1. bosh ssh into the master VM from the TKGi cluster

bosh -d service-instance_<GUID> ssh master/0
sudo -i

2. load the variables required to send NSX api calls

source /var/vcap/jobs/pks-nsx-t-prepare-master-vm/bin/pre-start

3. backup the old certificate <old_certificate_id>

curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/trust-management/certificates/<certificate_id>"  --cert ${NSX_MANAGER_CLIENT_CERT_FILE}  --key ${NSX_MANAGER_CLIENT_KEY_FILE}  --cacert ${NSX_MANAGER_CA_CERT_FILE}  -H "accept: application/json"  -H "Content-Type: application/json" > backup_certificate.json

4. list all virtual-server

curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/loadbalancer/virtual-servers/" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE}

5. backup the virtual server <virtual_server_id> that is still referencing the old certificate

curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/loadbalancer/virtual-servers/<virtual_server_id>" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} > backup_virtual-server.json
cp backup_virtual-server.json patch_virtual-server.json

6. Delete the old certificate id in the file in section "client_ssl_profile_binding" in the file new_virtual-server.json then patch the virtual server

curl -X PUT "https://${NSX_MANAGER_HOST}/api/v1/loadbalancer/virtual-servers/<virtual_server_id>" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} -H "content-type:application/json" -d @patch_virtual-server.json

7. Delete the old certificate

curl -X DELETE "https://${NSX_MANAGER_HOST}/api/v1/trust-management/certificates/<old_certificate_id>" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --ke ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} -H "X-Allow-Overwrite: true"