Workload Cluster stuck in Deleting state due to "paused: true" variable in Cluster Manifest

Products

• VMware Tanzu Mission Control
• VMware Tanzu Kubernetes Grid Service (TKGs)
• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid

Issue/Introduction

• After attempting deletion of a workload cluster, the cluster status is stuck in "Deleting" state.
• The cluster health shows "Unknown"
• VM's associated with the cluster are still present in the backing cloud provider.
• Attempting deletion from the CLI using kubectl commands also hangs and fails to delete the cluster

Environment

This might occur on TKGS or TKGm backing clusters operated by the Cluster API

Cause

If the cluster is configured with "paused: true", it will not be operated against. The Paused flag must be removed in order to allow the CAPI controllers to delete the cluster.

 

To determine if the cluster is paused, use the following command which will return "paused: true" :

kubectl get cluster <CLUSTER_NAME> -n NAMESPACE_NAME -o yaml | grep -i pause

Resolution

WARNING: This operation is only supported on clusters that are in Deleting state. These steps should not be performed on upgrading clusters.

If these unpause steps are performed on an upgrading or migrating cluster, the cluster may not be recoverable and will need to be redeployed.

 

Use the following command to remove the "paused: true" flag from the cluster to allow remediation:

kubectl patch cluster <CLUSTER_NAME> -n <NAMESPACE_NAME> --type merge -p '{"spec":{"paused": false}}'

 

Example:

kubectl patch cluster test-cluster -n test-cluster-namespace --type merge -p '{"spec":{"paused": false}}'