This is a known issue impacting the cluster created using Container Service Extension (CSE) up to and including version 4.2, and the Container Storage Interface (CSI) driver up to and including version 1.5.
As the current version of the CSI driver currently does not support resizing of PVCs while attached to a Pod, see the workaround section below.
Workaround:
NOTE:
Before proceeding you will need a KUBECONFIG file to access the affected cluster and VCD credentials for the cluster author.
See Manage Clusters for more information regarding obtaining a copy of the current Kube Config.
The high-level process includes four phases:
Identify which Deployment or StatefulSet controls the affected Pod. You will use this resource to control the affected Pod.
The Pod must be shutdown before making any changes.
If the controlling resource is managed by kapp-controller, the PackageInstall object must be paused or changes to the resource will be automatically overwritten.
The Pods for the resource may now be terminated by scaling it down to zero replicas.
Use kubectl to retrieve information about the affected PVC.
Record the NAME and VOLUME of the affected PVC for later steps.
Note:
This process may need to be repeated multiple times if there are multiple replicas in the StatefulSet.
Repeat this process for each Named Disk before continuing to the next step.
This process will use the VCD UI to resize the Named Disk associated with the PVC.
The underlying volume for the PVC has now been increased in size, but the filesystem has not been expanded.
Note:
This process may need to be repeated multiple times if there are multiple replicas in the StatefulSet.
Repeat this process for each PVC before continuing to the next step.
This process will use a temporary Pod to mount the PVC so you may resize the filesystem to consume the expanded capacity.
The prompt will pause while the Pod is scheduled and started.
Click Enter a couple of times if you think it is ready but don’t see a command prompt.
Run df /mnt to identify the device associated with the mounted PVC. Record the value of Filesystem.
Run resize2fs to resize the filesystem to consume the expanded capacity
Exit the shell. The Pod will be removed.
The filesystem on the PVC has now been updated to consume the expanded capacity of the underlying volume.
If the controlling resource is managed by kapp-controller, then you can unpause the PackageInstall.
The package will reconcile and update the resource to the desired number of replicas.
Otherwise, use kubectl to scale the resource back to the initial number of replicas.
Monitor the Pod status to ensure they start.
Restart the troubleshooting process if they continue to fail.