Error: "The object 'vim.Task:<task-id>' has already been deleted or has not been completely created" while configuring Supervisor Cluster host nodes

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Supervisor Cluster in a vSphere with Tanzu environment becomes stuck in the Configuring state
• This behavior is often accompanied by a general system error related to a task that no longer exists or has not been completely created on the ESXi host
• Symptoms can be observed as below: 
  • The Supervisor Cluster is stuck in the Configuring state within the vSphere Client 

  • Kubernetes Worker Node is schedulable: A general system error occurred. Error message: waiting for node <node-name> to move to ready state.
    
    Kubernetes Worker Node commit work done in vSphere: A general system error occurred. Error message: ServerFaultCode: The object 'vim.Task:<task-id>' has already been deleted or has not been completely created.

• The related ESXi task cannot be found using vim-cmd vimsvc/task_info <task-id> and returns a ManagedObjectNotFound
• kubectl get nodes -A confirms that all worker nodes are in a Ready state, despite the Supervisor Cluster showing otherwise.

Environment

VMware vSphere Kubernetes Service

VMware vSphere with Tanzu

vCenter Server 8.0.x

Cause

• This issue occurs when a stale task reference remains within the ESXi host’s management layer (hostd).
• The stale task causes vCenter Server to incorrectly believe that a Kubernetes worker node is still transitioning to a Ready state, even though the node is already healthy.
• This condition is typically triggered by interrupted or incomplete task cleanup in ESXi.

Resolution

1. To resolve the issue, connect to the affected ESXi host via SSH and restart the hostd management service on the affected ESXi host to clear stale task entries 

   /etc/init.d/hostd restart

2. Wait for the service to restart successfully
3. Verify that the stale task is cleared:
   vim-cmd vimsvc/task_list
   
   
4. Ensure that the previously reported task ID (e.g., task-#####) no longer appears
5. Recheck the Supervisor Cluster status in vCenter Server.
   It should now transition from Configuring to Running.
6. Note: If the task is not showing when command is run from step 3 and there are still issues in the environment, proceed with the steps below:
   1. Place the ESXi host in maintenance mode 
   2. Configure Host nodes
   3. Once configuration has been completed navigate back to the inventory and exit the ESXi host from maintenance mode
   4. Once out of the maintenance mode reconfigure the ESXi host node from the Workload Management