How storage is allocated to the kubelet and containerd in TKGI
Article ID: 390232
Updated On: 03-28-2025
Products
Issue/Introduction
This article a starting point for TKGI administrators to begin troubleshooting storage related issues with the kubelet and containerd components, specifically within TKGI.
The topics cover issues such as:
- Pods being evicted
- Images being garbage collected
Topics:
- Which specific filesystem identifiers TKGI uses for kubelet filesystems
- The persistent storage mount point used by both kubelet and containerd
- What determines persistent storage size?
- The ephemeral storage mount point used
- What determines ephemeral storage size?
- The imagefs filesystem/path where container images are stored
- Commands for seeing these details on worker nodes
Environment
TKGI
Bosh
Tanzu Operations Manager (Opsmanager)
Resolution
- Which specific filesystem identifiers TKGI uses for kubelet filesystems
In TKGI the kubelet utilizes two filesystems for the ("split disk"):
-
- An ephemeral disk (nodefs) on worker VMs dedicated for running pod containers. (/var/vcap/data)
- Container images (imagefs) on worker VMs are stored on persistent disk. (/var/vcap/store)
- An ephemeral disk (nodefs) on worker VMs dedicated for running pod containers. (/var/vcap/data)
Refer to the following for details:
Filesystem signals (nodefs and imagefs) for more information.
- The persistent storage mount point used by both kubelet and containerd
/var/vcap/store
- What determines persistent storage size?
Persistent Disk (/var/vcap/store) on worker nodes is configured in Opsmanager UI -> TKGI Tile -> PlanX -> "Worker Persistent Disk Type"
- The ephemeral storage mount point used
/var/vcap/data
- What determines ephemeral storage size?
The Ephemeral Disk (/var/vcap/data) size is configured in Opsmanager UI -> TKGI Tile -> PlanX -> "Worker VM Type"
- The imagefs filesystem/path where container images are stored
/var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs
- Commands for seeing the above details on worker nodes
To see the interworking of this:
Connect to a worker node and become root:
$ bosh ssh -d service-instance_CLUSTER_ID ssh worker/XXXX
worker/XXXX:~$ sudo -i
The ephemeral filesystem (nodefs) is the Filesystem Size of the /var/vcap/data mount point on the worker nodes
And
The persistent filesystem (imagefs) is the Filesystem Size of the /var/vcap/store mount point on the worker nodes
worker/XXXX:~# df -h | egrep 'Filesystem|^/dev'Filesystem Size Used Avail Use% Mounted on/dev/sda2 4.8G 2.7G 1.9G 60% //dev/sda1 48M 5.7M 43M 12% /boot/efi/dev/sdb1 32G 5.2G 25G 18% /var/vcap/data/dev/sdc1 49G 3.8G 43G 9% /var/vcap/store
To show where the containerd image filesystem(imagefs) is located:
worker/XXXX:~# crictl imagefsinfo{ "status": { "timestamp": "1741377915871870723", "fsId": { "mountpoint": "/var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs" }, "usedBytes": { "value": "2063876096" }, "inodesUsed": { "value": "29521" } }}
Show total storage used by the container images (imagefs). Example:
worker/XXXX:~# du -sh /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs
2.0G /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs
Compare the above to the storage info for the persistent filesystem :
worker/XXXX:~# df -h /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfsFilesystem Size Used Avail Use% Mounted on/dev/sdc1 49G 3.8G 43G 9% /var/vcap/store
Kubelet customization references:
Refer to "Kubelet customization - eviction-hard" in the Plans section of Installing TKGI on vSphere
Other references:
Feedback
