PKS cluster deletion fails with error "Deployment ' ' doesn't exist"

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

If the cluster creation fails due to a network configuration error, such as unresponsive agent on master vm, while creating a cluster using the command, pks create-cluster, the deployment is left in an inconsistent state.

Cluster deletion through the PKS CLI does not work in this scenario.

This article describes how the deployment can be deleted if the cluster ends up in this situation.

The following is the error message is produced when there is a network configuration issue:

Error: - Deployment '' doesn't exist

Environment

Cause

After the command, pks delete-cluster <cluster-name> has failed use the command, pks cluster <cluster-name> to identify the BOSH task.

From the BOSH task id identified above, confirm if it has the following error:

ubuntu@bosh-stemcell:~$ bosh task 1526
Using environment '172.###.###.###' as client 'ops_manager'

Task 1526

Task 1526 | 12:56:00 | Preparing deployment: Preparing deployment (00:00:00)
                    L Error: - Deployment '' doesn't exist
Task 1526 | 12:56:00 | Error: - Deployment '' doesn't exist

Task 1526 Started  Tue Jun 26 12:56:00 UTC 2018
Task 1526 Finished Tue Jun 26 12:56:00 UTC 2018
Task 1526 Duration 00:00:00
Task 1526 error

Capturing task '1526' output:
  Expected task '1526' to succeed but state is 'error'

Exit code 1

In this case, deployment was present but the K8s master virtual machine (VM) was deleted as it is in an unresponsive state.

ubuntu@bosh-stemcell:~$ bosh vms
Using environment '172.###.###.###' as client 'ops_manager'

Task 1527
Task 1528
Task 1527 done

Task 1528 done

Deployment 'pivotal-container-service-########e383eff77532'

Instance                                                        Process State  AZ   IPs          VM CID                                   VM Type  Active  
pivotal-container-service/5c1f8ac3-772c-4696-90d1-d55004934fa8  running        AZ1  172.###.###.###  vm-b21eff31-9a26-4943-9e38-af9dee6b8217  large    -  

1 vms

Deployment 'service-instance_########-63ad-4a0f-81ff-2da3842a2c89'

Instance                                     Process State  AZ   IPs         VM CID                                   VM Type  Active  
worker/3927eedf-e749-4fbb-952f-1491d0879fcd  running        AZ1  172.###.###.###  vm-aca5b05e-1215-4d03-9fb2-23c76b20bed9  medium   -  
worker/45a9af36-903c-42b3-9e27-ad78f08cfe67  running        AZ1  172.###.###.###  vm-e615c3b8-39e9-40a0-8573-0807ec3e6967  medium   -  

2 vms

Resolution

Follow the below steps to delete the deployment and remove it's entry from the PKS database:

Delete the BOSH deployment from the BOSH CLI using the following command:
```
bosh delete-deployment -d service-instance_#### --force
```
Use the PKS CLI to delete the cluster again:
```
pks delete-cluster <failed cluster name>
```

Additional Information

Note: The below steps are required only if the PKS version is less than v1.1.5.

SSH to the PKS VM and switch to root.

bosh ssh -d pivotal-container-service-####
sudo su -

Connect to the PKS database and remove the failed cluster entry from the cluster table.

mysql -h 127.0.0.1 -u root -p 

Enter password:  Use password from PKS tile -> Credentials -> Cf Mysql Mysql Admin Password

MariaDB [pks]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| pks                |
| telemetry          |
| uaa                |
+--------------------+
6 rows in set (0.01 sec)

MariaDB [(none)]> use pks

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [pks]> show tables;
+-----------------------+
| Tables_in_pks         |
+-----------------------+
| cluster               |
| flyway_schema_history |
+-----------------------+
2 rows in set (0.00 sec)

MariaDB [pks]> delete from cluster where name = <failed cluster name> ;