TKGI v1.23.0 upgrade leads to changes in the imagefs directory that can cause disk pressure and pod eviction

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In TKGI v1.23.0, the directory for imagefs has changed.  When upgrading from v1.22.x to v1.23.0, the old imagefs directory does not get deleted. This results in the old directory persisting as it was, and the new directory is loaded with data and images, which effectively will double the disk usage unnecessarily. This potentially can create disk pressure issue which could result in pod eviction.

Environment

TKGi 1.23.0

Cause

The imagefs directory location has changed in TKGI v1.23.0.

To check the directory location, run command crictl imagefsinfo on a worker VM:

TKGI v1.22.x

crictl imagefsinfo
{
  "status": {
    "timestamp": "1762182059818853993",
    "fsId": {
      "mountpoint": "/var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs"
    },

TKGI v1.23.0 

 crictl imagefsinfo
{
  "status": {
    "timestamp": "1762264051739809869",
    "fsId": {
      "mountpoint": "/var/vcap/store/io.containerd.snapshotter.v1.overlayfs"
    },

 

Resolution

Scenario 1: If you have already upgraded a cluster to TKGI v1.23

Refer to the TKGI v1.23.0 changes imagefs directory KB.

Scenario 2: If you have upgraded to TKGI tile to v1.23 but the clusters are pending upgrade (not yet upgraded), proceed with applying the 'os-conf' patch and then perform the upgrade.

For additional information about how to apply the 'os-conf' patch, refer to the How to use BOSH os-conf release to run script at different deployment stage KB.

Example of Runtime Config setup for targeting specific service instance before upgrade:

Create a runtime.yml file:

releases:
- name: "os-conf"
  version: "23.0.0"
addons:
- name: containerd-configuration
  jobs:
  - name: pre-start-script
    release: os-conf
    properties:
      script: |-
        #!/bin/bash
        CONTAINERD_CONFIG=/var/vcap/jobs/containerd/config/config.toml
        if [ -e "$CONTAINERD_CONFIG" ]; then
          echo "Changing root dir of containerd"
          sudo sed -i '2c\root="/var/vcap/store/containerd"' $CONTAINERD_CONFIG
          echo "Done"
        fi
  include:
    deployments: [service-instance_XXX, service-instance_YYY]    # Optional, you can define which deployments (TKGi clusters) this runtime config will be applied to.
    instance_groups: [worker]

NOTE: If you don't specify any deployments, then the runtime-config will be applied to any deployment including new clusters.

Additional Information

Expected Fix in TKGi 1.23.1 

The new patch will restore the location of the folder to original path: /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs

Therefore prior upgrade to 1.23.1 delete the os-conf created.