tkgi delete-cluster fails, cluster VMs unresponsive

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

An attempt to delete a TKGi cluster using "tkgi delete-cluster" fails with error

"Error: Error on request to OD Service Broker"

The cluster VMs are all unresponsive and Bosh director times out getting the status of the VMs

ERROR -- DirectorJobRunner: Worker thread raised exception: master/########-####-####-####-##########: Timed out sending 'get_state' to instance: 'master/#########-####-####-####-##########', agent-id: '########-####-####-####-#############' after 45 seconds - 

Environment

TKGi

Resolution

Run bosh cck to delete the VMs, select option 5

bosh -d service-instance_<GUID> cck

1: Skip for now
2: Reboot VM
3: Recreate VM without waiting for processes to start
4: Recreate VM and wait for processes to start
5: Delete VM
6: Delete VM reference (forceful; may need to manually delete VM from the Cloud to avoid IP conflicts)

Confirm VMs are deleted

bosh -d service-instance_<GUID> vms

Run "tkgi delete-cluster" again and confirm its successful

tkgi delete-cluster <Cluster name>
tkgi clusters