Cannot delete an Organization on VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

1. When deleting an Organization on VMware Cloud Director (VCD) you receive the error below:
   [ xxxx-xxxx-xxxx-xxxx-xxxx ]  ERROR: update or delete on table "org_member" violates foreign key constraint "fk_custom_entity2org_member" on table "defined_entity"  Detail: Key (org_member_id)=(xxxx-xxxx-xxxx-xxxx-xxxx) is still referenced from table "defined_entity".


2. From /opt/vmware/vcloud-director/logs/vcloud-container-debug.log you have entries similar below:
   
   YYYY-MM-DD TIME | ERROR    | task-service-activity-pool-3055 | JDBCExceptionReporter          | ERROR: update or delete on table "org_member" violates foreign key constraint "fk_custom_entity2org_member" on table "defined_entity" Detail: Key (org_member_id)=(xxxx-xxxx-xxxx-xxxx-xxxx) is still referenced from table "defined_entity". | requestId=xxxx-xxxx-xxxx-xxxx-xxxx,request=DELETE https://vcloud.example.com/cloudapi/1.0.0/orgs/urn:vcloud:org:xxxx-xxxx-xxxx-xxxx-xxxx,requestTime=1720779150377,remoteAddress=##.##.##.##:56226,userAgent=Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:109.0) Gecko/2010...,accept=application/json;version 39.0.0-alpha vcd=xxxx-xxxx-xxxx-xxxx-xxxx,task=xxxx-xxxx-xxxx-xxxx-xxxx activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:xxxx-xxxx-xxxx-xxxx-xxxx)

3. In the VCD task you can see a CSE cluster was not correctly deleted and reporting the following error:
   
   error deleting resources by rdeId: [urn:vcloud:entity:vmware:capvcdCluster:xxxx-xxxx-xxxx-xxxx-xxxx], [error deleting resources by rde ID: [urn:vcloud:entity:vmware:capvcdCluster:xxxx-xxxx-xxxx-xxxx-xxxx] after [10] retry attempts: [error occurred deleting L4 loadbalancer for rde [podinfo-test(urn:vcloud:entity:vmware:capvcdCluster:xxxx-xxxx-xxxx-xxxx-xxxx)]: [virtual service [podinfo-test-urn:vcloud:entity:vmware:capvcdCluster:xxxx-xxxx-xxxx-xxxx-xxxx-tcp] is busy]. [0] remaining retry attempts]]
   
   

Environment

VMware Cloud Director 10.5.x

Cause

There is an Organization user that still owns a CSE cluster that was not correctly deleted.

Resolution

Delete the CSE cluster following the KB Cannot delete Kubernetes Container cluster within Cloud Director

Delete the organization form the VCD UI

 

To check the CSE cluster info:

1- Note the entity ID from the error:

[ xxxx-xxxx-xxxx-xxxx-xxxx ]  ERROR: update or delete on table "org_member" violates foreign key constraint "fk_custom_entity2org_member" on table "defined_entity"  Detail: Key (org_member_id)=(ENTITY_ID) is still referenced from table "defined_entity".

2- Connect to VCD DB on the primary cell:

sudo -i -u postgres psql vcloud

3- Run the below query:

select * from defined_entity where owner_id = 'ENTITY_ID_from_the_error_above';

4- Once you have the name of  the CSE cluster from point 3, you can check in the VCD events if there is the failed deletion task for the above cluster.