Automated Workload Domain Deletion Fails with TA_TASK_NOT_FOUND and No Default Cluster Found Error

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Automated Workload Domain (WLD) deletion fails in VMware Cloud Foundation (VCF).
• When the automation queries the generated Task ID, the API returns a TA_TASK_NOT_FOUND error. Furthermore, the SDDC Manager UI does not track or report any in-progress or failed domain deletion tasks.
• A review of the /var/log/vmware/vcf/domainmanager/domainmanager.log file reveals the following error:

YYYY:MM:DDTHH:MM:SS.Z DEBUG [vcf_dm,69e5bcaf488c1bcaf9baea40ccf6e79b,f541] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7200-exec-6]  Fetching NSX Cluster from inventory for domain ID 7#######-####-####-####-############
YYYY:MM:DDTHH:MM:SS.Z DEBUG [vcf_dm,69e5bcaf488c1bcaf9baea40ccf6e79b,f541] [c.v.e.s.c.s.NsxtVersionServiceImpl,http-nio-127.0.0.1-7200-exec-6]  SubTnp feature is available in NSX-T 4.2.1.3.0-24533884
YYYY:MM:DDTHH:MM:SS.Z DEBUG [vcf_dm,69e5bcaf488c1bcaf9baea40ccf6e79b,f541] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7200-exec-6]  Get Default Cluster by Domain - 7#######-####-####-####-############
YYYY:MM:DDTHH:MM:SS.Z ERROR [vcf_dm,69e5bcaf488c1bcaf9baea40ccf6e79b,f541] [c.v.e.s.c.s.a.w.o.WorkflowOptionsAdapterUtil,http-nio-127.0.0.1-7200-exec-6]  Error reading inventory for NSX transport zones information for domain 7#######-####-####-####-############
com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: No default cluster found for domain : 7#######-####-####-####-############
        at com.vmware.evo.sddc.common.services.adapters.inventoryservice.InventoryServiceAdapterImpl.lambda$getDefaultClusterByDomainId$45(InventoryServiceAdapterImpl.java:205
5)
        at java.base/java.util.Optional.orElseThrow(Optional.java:403)
        at com.vmware.evo.sddc.common.services.adapters.inventoryservice.InventoryServiceAdapterImpl.getDefaultClusterByDomainId(InventoryServiceAdapterImpl.java:2053)
        at com.vmware.evo.sddc.common.services.adapters.workflow.options.WorkflowOptionsAdapterUtil.isVlanBackedNsxDomain(WorkflowOptionsAdapterUtil.java:1500)
        at com.vmware.evo.sddc.common.services.adapters.workflow.options.WorkflowOptionsAdapterImpl.getWorkflowOptionsForRemoveDomain(WorkflowOpt

• Querying the cluster table in the platform database shows the target cluster  has the is_default flag set to f (false).

select id, is_default, name from cluster; 

id                                    is_default  name
a#######-####-####-####-############  f           <cluster_name>

Environment

VCF 5.x 

VC 9.0.x 

Cause

The cluster within the Workload Domain has its is_default database flag set to false. During the WLD deletion workflow, SDDC Manager attempts to load the NSX inventory, which explicitly requires locating the default cluster of the domain. Because no cluster in the database is flagged as the default(is_default='t') for the domain, the validation step fails, preventing the task from executing in SDDC Manager.

Resolution

• Take a snapshot of the SDDC Manager appliance or perform a full backup to an external SFTP server.

• Log in to the SDDC Manager via SSH.

• Elevate to the root account using the following command: su root

• Connect to the platform database.

• Run the following query to mark the domain's cluster as true for its default status:

update cluster set is_default='t' where name='<cluster_name>'

• Exit the database using the following command: \q

• Restart the domain manager service: systemctl restart domainmanager

• Retry the Workload Domain deletion workflow.