How storage is allocated to the kubelet and containerd in TKGI
search cancel

How storage is allocated to the kubelet and containerd in TKGI

book

Article ID: 390232

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition VMware Tanzu Platform - Cloud Foundry VMware Tanzu Platform - Kubernetes

Issue/Introduction

This article a starting point for TKGI administrators to begin troubleshooting storage related issues with the kubelet and containerd components, specifically within TKGI.  

The topics cover issues such as:

- Pods being evicted

- Images being garbage collected

 

Topics:

- Which specific filesystem identifiers TKGI uses for kubelet filesystems

- The persistent storage mount point used by both kubelet and containerd

- What determines persistent storage size?

- The ephemeral storage mount point used

- What determines ephemeral storage size?

- The imagefs filesystem/path where container images are stored

- Commands for seeing these details on worker nodes

Environment

TKGI

Bosh

Tanzu Operations Manager (Opsmanager)

Resolution

- Which specific filesystem identifiers TKGI uses for kubelet filesystems

In TKGI the kubelet utilizes two filesystems for the  ("split disk"):

    1. An ephemeral disk (nodefs) on worker VMs dedicated for running pod containers. (/var/vcap/data)

    2. Container images (imagefs) on worker VMs are stored on persistent disk. (/var/vcap/store)

Refer to the following for details:

Filesystem signals (nodefs and imagefs) for more information.

With imagefs

 

- The persistent storage mount point used by both kubelet and containerd

/var/vcap/store

 

- What determines persistent storage size?

Persistent Disk (/var/vcap/store) on worker nodes is configured in Opsmanager UI -> TKGI Tile -> PlanX -> "Worker Persistent Disk Type"

 

- The ephemeral storage mount point used

/var/vcap/data

 

- What determines ephemeral storage size?

The Ephemeral Disk (/var/vcap/data) size is configured in Opsmanager UI -> TKGI Tile -> PlanX -> "Worker VM Type"

 

- The imagefs filesystem/path where container images are stored

/var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs

 

- Commands for seeing the above details on worker nodes

To see the interworking of this:

Connect to a worker node and become root:

$ bosh ssh -d service-instance_CLUSTER_ID ssh worker/XXXX

worker/XXXX:~$ sudo -i

 

The ephemeral filesystem (nodefs) is the Filesystem Size of the /var/vcap/data mount point on the worker nodes

And

The persistent filesystem (imagefs) is the Filesystem Size of the /var/vcap/store mount point on the worker nodes

worker/XXXX:~# df -h | egrep 'Filesystem|^/dev'
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       4.8G  2.7G  1.9G  60% /
/dev/sda1        48M  5.7M   43M  12% /boot/efi
/dev/sdb1        32G  5.2G   25G  18% /var/vcap/data
/dev/sdc1        49G  3.8G   43G   9% /var/vcap/store

 

To show where the containerd image filesystem(imagefs) is located:

worker/XXXX:~# crictl imagefsinfo
{
  "status": {
    "timestamp": "1741377915871870723",
    "fsId": {
      "mountpoint": "/var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs"
    },
    "usedBytes": {
      "value": "2063876096"
    },
    "inodesUsed": {
      "value": "29521"
    }
  }
}

 

Show total storage used by the container images (imagefs).  Example:

worker/XXXX:~#  du -sh /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs

2.0G    /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs

 

Compare the above to the storage info for the persistent filesystem :

worker/XXXX:~# df -h /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1        49G  3.8G   43G   9% /var/vcap/store

 

 

 

Kubelet customization references:

Refer to "Kubelet customization - eviction-hard" in the Plans section of Installing TKGI on vSphere


Other references:

https://containerd.io/docs/