Deleting an Enterprise PKS cluster may not complete when the vSphere CSI is used to provision persistent volumes

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

• A cluster is created in Enterprise PKS 1.7 and vSphere CSI is configured to allow for persistent volumes to be created via vSphere Cloud Native Storage (CNS).
• A workload is deployed with persistent volumes that makes use of vSphere CNS.
• An attempt to delete the cluster is made but it hangs indefinitely.
• You see that the drain-cluster errand task is not completing. You see output similar to the following when running bosh task <id>:

Task 137 | 13:17:33 | Preparing deployment: Preparing deployment
Task 137 | 13:17:33 | Deprecation: Global 'properties' are deprecated. Please define 'properties' at the job level.
Task 137 | 13:17:36 | Warning: Executing errand on multiple instances in parallel. Use the `--instance` flag to run the errand on a single instance.
Task 137 | 13:17:36 | Preparing deployment: Preparing deployment (00:00:03)
Task 137 | 13:17:36 | Running errand: worker/64b96419-83bb-48eb-a1c5-5823855df887 (1)
Task 137 | 13:17:36 | Running errand: worker/f85cef70-0dc8-41ba-9a47-5c64c844411b (0)
Task 137 | 13:17:40 | Running errand: worker/64b96419-83bb-48eb-a1c5-5823855df887 (1) (00:00:04)
Task 137 | 13:17:40 | Fetching logs for worker/64b96419-83bb-48eb-a1c5-5823855df887 (1): Finding and packing log files (00:00:01)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Native Storage 1.x
VMware PKS 1.7.x

Resolution

This is a known issue affecting Enterprise PKS 1.7 when vSphere Cloud Native storage is used. There is currently no resolution. 

Workaround:
To workaround this issue you can issue the bosh cancel-task <id> command to cancel the drain-cluster errand task.

To prevent this issue from occurring you can manually delete the workload that is using CNS to provision persistent volumes prior to deleting the cluster.

Additional Information

Cloud Native Storage (CNS) on vSphere