Creating TKGs guest cluster on TMC Stuck when trying to expand storage capacity

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

While using TMC UI to create a new guest cluster on vSphere with Tanzu. Guest cluster stuck on running status on the TMC UI 

On vSphere side, just one control plane is created and nothing more is up.

From cloud-init-output.log:

umount /var/lib
umount: /var/lib: target is busy.

 

Cause

On TMC UI , While creating the TKC there is an option to add a disk to the control planes and the worker nodes.

Issue happened when tried to add disk on /var/lib directory.

TKC spec is attempting to expand storage capacity for the entire `/var/lib` directory, but data migration fails - `target is busy` means a process has a file open somewhere in the `/var/lib` directory.

Resolution

Solution is to use the following subdirectories instead of using /var/lib


For control plane nodes:

Mount Path:
  /var/lib/etcd

For worker nodes:

Mount Path: 
 /var/lib/containerd

From https://tanzu.vmware.com/content/blog/deploy-tanzu-kubernetes-clusters-data-volumes-tanzu-mission-control 

Additional Information

Impact/Risks:
TKC creation is stuck