CCI and Velero Supervisor Services Deployment Failure Due to Full Root Partition on Control Plane VMs
Article ID: 443301
Updated On:
Products
Issue/Introduction
When attempting to deploy or update Supervisor services such as CCI (Cloud Consumption Interface) or Velero vSphere Operator , the following symptoms are observed:
- CCI Pods: Fail to reconcile with `ProgressDeadlineExceeded` or remain in a constant recreation loop.
- Velero Pods: Stuck in `Pending` or `ImagePullBackOff`.
- Resource Status: `kubectl get imagedisks` shows disks stuck in an `Allocating` state.
- vSphere Client: Supervisor cluster shows a `Configuring` or `ReconcileFailed` status with messages like:
- “Deployment is not progressing: ProgressDeadlineExceeded”
- “podVMcreationFailed”
- Node Status: One or more Supervisor Control Plane VMs may show as `NotReady`
Environment
vSphere with Tanzu, vSphere Kubernetes Service (VKS)
vCenter 8.0.3, Supervisor v1.29.7+
Cause
The primary cause is the **root (/) filesystem** on the Supervisor Control Plane VMs reaching 100% (or exceeding the 80% threshold). This prevents the system from:
- **Image Handling:** Writing temporary data during the image resolution and unpacking process.
- **Service Stability:** Critical processes like the `kube-apiserver`, `etcd`, and `image-controller` begin to crash or lose quorum when they cannot write logs or snapshots to disk.
- **PodVM Creation:** The `image-fetcher` service cannot allocate disks for new PodVMs when the local cache/temp directory on the root partition is full.
Common contributors to disk bloat include unrotated `upgrade-ctl-cli.log` files, accumulated `journalctl` logs, and stale images from previous upgrades.
Resolution
Follow these steps to clear space on each of the three Supervisor Control Plane VMs:
1. Reclaim space on the root filesystem following KB 383369: vSphere Supervisor Root Disk Space Full at 100%
- Clear Journal Logs
- Remove rotated historical upgrade logs that can grow to 1GB each
- Clean Stale Kubernetes Resources
In a Supervisor context
2. In line with KB 389539: Supervisor Services failed to get images when deploying, . Delete the consumption interface pods. Redeploying the pods should force the system to fetch the image afresh, bypassing the "resolving" state issue.
3. In line with KB 389539: Supervisor Services failed to get images when deploying, Identify any imagedisk that is not Ready. The "allocating" or "resolving" states typically indicate an issue with the image resolution process.
kubectl get imagedisks -A
kubectl get images -A
4. In line with KB 389539: Supervisor Services failed to get images when deploying, Delete the imagedisks and their images where the imagedisk is not in a Ready state:
kubectl delete -n vmware-system-kubeimage <imagedisk name>
kubectl delete -n <image namepace> <image name>
5. On the vCenter, restart the wcp service, this will force a Supervisor reconcile and the Supervisor Services should now be reconciled correctly:
vmon-cli -r wcp
Additional Information
Related Articles
KB 383369: vSphere Supervisor Root Disk Space Full at 100%
KB 389539: Supervisor Services failed to get images when deploying
Feedback
