Unable to delete the cluster of a Workload Domain in SDDC Manager

Products

• VMware Cloud Foundation
• VMware SDDC Manager

Issue/Introduction

• Unable to delete the default Workload Cluster from the SDDC Manager GUI or via API
• Delete option is unavailable or disabled/"greyed out" for the Cluster
• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log when attempted to delete the cluster via API

  DEBUG [vcf_dm,68c8...................] [c.v.v.c.f.a.RemoveClusterGetInventoryAction,dm-exec-20]  Validate not default cluster
  ERROR [vcf_dm,68c8...................] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-20]  [#####] DEFAULT_CLUSTER Trying to remove default cluster <cluster-name_to_be_deleted> 
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Trying to remove default cluster <cluster-name_to_be_deleted>

Environment

• VMware Cloud Foundation
• VMware SDDC Manager

Cause

The default Workload cluster of a domain cannot be removed.

Removal of MGMT Domain (and its cluster) is also not allowed.

Resolution

Note: This is a destructive operation. Please ensure that all necessary backups and snapshots are taken prior to proceeding with the steps below to avoid any issue.

To be able to delete the current default cluster of a Workload Domain, we have to set the cluster in the Workload Domain as not default, so that the cluster delete option is available in UI or API. 

1. Take a snapshot of the SDDC Manager VM
2. SSH into the SDDC Manager VM with the vcf user, then elevate to root (su root)
3. Fetch the Cluster details in SDDC Manager database where is_default is marked as true

   curl 127.0.0.1/inventory/clusters | json_pp

   Sample output

   [
      {
         "datacenterSourceId" : "datacenter-3",
         "domainId" : "<domain-id>",
         "id" : "<cluster-id_to_be_deleted>",
         "isDefault" : true,  ==================> This shows cluster is marked as default
         "isImageBased" : true,
         "isImported" : false,
         "isStretched" : false,
         "primaryDatastoreSourceId" : "datastore-##",
         "primaryDatastoreType" : "NFS",
         "sourceId" : "domain-c##",
         "status" : "ACTIVE",
         "vcenterId" : "<vcenter-id>",
         "vdsIds" : [
            "<vds-id>"
         ]
      },
   ]

   
   If Cluster name is not populated in the CURL command then fetch the cluster ID directly from database

   psql -h localhost -U postgres -d platform -c "select id,name,is_default from cluster"

   Sample output

                 id                  |     name      | is_default
   --------------------------------------+---------------+------------
    <cluster-id_to_be_deleted> 		  | wld01cluster1 | t
    <mgmt_cluster_id>		  | mgmtdom-cl01  | t
    <cluster02_ID>			  | wld01cluster2 | f

4. Execute the below command to fetch the cluster details in a json file

   curl 127.0.0.1/inventory/clusters?id=<cluster-id_to_be_deleted> | json_pp > wld01cluster1.json

5. Update the json file to change default state to false

   vi  wld01cluster1.json

6. Remove the Opening and Closing square bracket and change "isDefault" to false
   
   sample updated json

   {
         "datacenterSourceId" : "datacenter-3",
         "domainId" : "<domain-id>",
         "id" : "<cluster-id_to_be_deleted>",
         "isDefault" : false,  ==================> Cluster default status is false
         "isImageBased" : true,
         "isImported" : false,
         "isStretched" : false,
         "primaryDatastoreSourceId" : "datastore-##",
         "primaryDatastoreType" : "NFS",
         "sourceId" : "domain-c##",
         "status" : "ACTIVE",
         "vcenterId" : "<vcenter-id>",
         "vdsIds" : [
            "<vds-id>"
         ]
      },

7. Save the file

   Esc > :wq!

   
   Note: No other changes/updates to be done in the json
   
   
8. Give updated json as input to below API : This will update the entry inside 'cluster' table

   curl -s -X PUT "http://localhost/inventory/clusters/<cluster-id_to_be_deleted>" -H "Content-Type: application/json" -d @wld01cluster1.json

9. Mark another cluster in same Workload domain as default. (The default cluster is crucial for workload domain functionality. If not explicitly marked, it can cause creation failures)
   1. Execute the below command to fetch the cluster02  details in a json file

      curl 127.0.0.1/inventory/clusters?id=<cluster02_ID> | json_pp > wld01cluster02.json

   2. Update the json file to change default state to true

      vi wld01cluster02.json

   3. Remove the Opening and Closing square bracket and change "isDefault" to true

       {
            "datacenterSourceId" : "datacenter-3",
            "domainId" : "<domain-id>",
            "id" : "<cluster02_ID>",
            "isDefault" : true,  ==================> Cluster default status is true
            "isImageBased" : true,
            "isImported" : false,
            "isStretched" : false,
            "primaryDatastoreSourceId" : "datastore-##",
            "primaryDatastoreType" : "NFS",
            "sourceId" : "domain-c##",
            "status" : "ACTIVE",
            "vcenterId" : "<vcenter-id>",
            "vdsIds" : [
               "<vds-id>"
            ]
         },

   4. Save the file

      Esc>:wq!

      Note: No other changes/updates to be done in the json

   5. Give updated json as input to below API : This will update the entry inside 'cluster' table

      curl -s -X PUT "http://localhost/inventory/clusters/<cluster02_ID>" -H "Content-Type: application/json" -d @wld01cluster02.json

10. Restart the domainmanager service

    systemctl restart domainmanager

11. Attempt the Cluster delete workflow from the SDDC Manager UI or via API.