Mitigating Kubelet Garbage Collection and Eviction Race Conditions in TKGI

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Kubelet Image Garbage Collection (GC) deletes critical system container images (e.g., CoreDNS, CSI syncers, Oratos telemetry) when worker node disk utilization reaches 85% (imagefs.available < 15%). The workaround described in KB "Core Images Deleted by Garbage Collector Are Not Reloaded in TKGI Air-Gapped Environment" involves increasing evictionHard.imagefs.available to 20% to force early workload eviction and trigger recovery scripts. However, this high eviction threshold causes premature and frequent pod evictions during normal cluster operations. This article details an alternative solution for environments where premature pod eviction is unacceptable. Implement this method to safely protect system images while allowing native garbage collection to run without disrupting active workloads.

Environment

VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)

Cause

A race condition exists between Kubernetes Garbage Collection thresholds and Eviction thresholds. When system images are not pinned in the containerd cache, Kubelet treats them as unused and deletes them to reclaim space.

Resolution

Resolving this issue is a two-phase process. Protect the system images using a lifecycle-aware pinning script first, and then tune the Kubelet thresholds to allow normal Garbage Collection to occur without triggering workload evictions.

Phase 1: Deploy the Lifecycle-Aware Pinning Script

Deploy the following bash script to the worker nodes (e.g., via a cron job running every 15-30 minutes). This script safely pins the active versions of critical system images to protect them from GC, while actively unpinning older versions to prevent persistent disk bloat during upgrade cycles.

#!/usr/bin/env bash
#
# TKGI Lifecycle-Aware System Image Pinning Script 
# Features: Hybrid SemVer/Metadata sorting, safe cross-registry mirror consolidation,
# and fail-safe checks for transient socket errors.

set -euo pipefail

CONTAINERD_BIN_DIR="/var/vcap/packages/containerd/bin"
SOCKET_PATH="/var/vcap/sys/run/containerd/containerd.sock"

CRICTL="${CONTAINERD_BIN_DIR}/crictl --runtime-endpoint unix://${SOCKET_PATH}"
CTR="${CONTAINERD_BIN_DIR}/ctr -a ${SOCKET_PATH} -n k8s.io"

PATH_PATTERNS="/(oratos|kas-network-proxy|csi-vsphere|antrea)/"
IMAGE_PATTERNS="/[^/]*(pause|coredns|metrics-server|nsx-node-agent|proxy-agent|telemetry-agent|cadvisor|syncer|snapshot-controller|snapshot-validation-webhook|vsphere-csi)[^/]*$"

if [ ! -f "${CONTAINERD_BIN_DIR}/crictl" ] || [ ! -f "${CONTAINERD_BIN_DIR}/ctr" ]; then
    echo "[ERROR] Containerd binaries not found at ${CONTAINERD_BIN_DIR}. Is this a TKGI worker VM?"
    exit 1
fi

echo "=================================================="
echo "Starting Hybrid Image Pinning Strategy (v13)"
echo "=================================================="

ALL_IMAGES=$(${CRICTL} images | tail -n +2 | awk '$2 != "<none>" {print $1, $2, $3}')
MATCHING_LINES=$(echo "$ALL_IMAGES" | grep -Ei "(${PATH_PATTERNS}|${IMAGE_PATTERNS})")

if [ -z "$MATCHING_LINES" ]; then
    echo "No matching system images found."
    exit 0
fi

FAMILY_LINES=$(echo "$MATCHING_LINES" | awk '{
    img=$1; tag=$2; id=$3;
    family=img;
    
    # 1. Strip registry domain for safe production mirror consolidation
    sub(/^[^/]+\//, "", family);
    
    # (Staging normalization removed to preserve independent fallback images)
    
    # 2. Strip embedded version strings for metrics-server
    if (family ~ /metrics-server/) {
        sub(/-v[0-9].*$/, "", family);
    }
    
    print family, img, tag, id
}')

FAMILIES=$(echo "$FAMILY_LINES" | awk '{print $1}' | sort -u)

for FAM in $FAMILIES; do
    echo "--------------------------------------------------"
    echo "Processing Family: ${FAM}"

    VARIANTS=$(echo "$FAMILY_LINES" | awk -v fam="$FAM" '$1 == fam {print $2, $3, $4}')
    WINNER_ID=""

    if [[ "$FAM" == "oratos/"* || "$FAM" == *"/oratos/"* ]]; then
        echo "    [STRATEGY] Hex Tags -> Using Metadata Timestamp"
        WINNER_ID=$(echo "$VARIANTS" | python3 -c '
import sys, json, subprocess
newest_time = ""
newest_id = ""
for line in sys.stdin:
    parts = line.strip().split()
    if len(parts) < 3: continue
    img_id = parts[2]
    try:
        cmd = ["/var/vcap/packages/containerd/bin/crictl", "--runtime-endpoint", "unix:///var/vcap/sys/run/containerd/containerd.sock", "inspecti", "-o", "json", img_id]
        output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
        data = json.loads(output)
        info_obj = data.get("info", {})
        created_at = ""
        if "info" in info_obj and isinstance(info_obj["info"], str):
            inner_data = json.loads(info_obj["info"])
            created_at = inner_data.get("imageSpec", {}).get("created", "")
        elif "imageSpec" in info_obj:
            created_at = info_obj.get("imageSpec", {}).get("created", "")
        
        if created_at > newest_time:
            newest_time = created_at
            newest_id = img_id
    except Exception:
        continue
print(newest_id)
')
    else
        echo "    [STRATEGY] Semantic Tags -> Using sort -V"
        WINNING_LINE=$(echo "$VARIANTS" | sort -V | tail -n 1)
        WINNER_ID=$(echo "$WINNING_LINE" | awk '{print $3}')
    fi

    if [ -z "$WINNER_ID" ]; then
        echo "    [WARNING] Could not determine a winner for ${FAM}. Skipping to prevent accidental unpinning."
        continue
    fi

    while read -r IMG TAG IMG_ID; do
        FULL_REF="${IMG}:${TAG}"
        if [ "$IMG_ID" == "$WINNER_ID" ]; then
            ${CTR} images label "${FULL_REF}" io.cri-containerd.pinned=pinned > /dev/null 2>&1
            echo "    [PINNED] ${FULL_REF}"
        else
            ${CTR} images label "${FULL_REF}" io.cri-containerd.pinned=unpinned > /dev/null 2>&1
            echo "    [UNPINNED] ${FULL_REF}"
        fi
    done <<< "$VARIANTS"
done

echo "=================================================="
echo "Lifecycle script execution complete."
echo "======================================

Phase 2: Tune Kubelet GC Thresholds (Choose Option A or B)

Now that the system images are protected by the script, safely tune the Kubelet to clean up standard user images gracefully.

Option A: Lower the evictionHard Threshold in Ops Manager

Instead of changing GC parameters, lower the eviction threshold so it triggers after normal GC.

Navigate to the Tanzu Kubernetes Grid Integrated Edition tile in Ops Manager.
Locate the plan you would like to update.
Under "Kubelet Customization - eviction hard", add or modify the "imagefs.available=XX%" flag, down to 10%.
Review the pending changes and apply them to the TKGi tile. To activate this configuration, you'll also need to upgrade the clusters where you'd like the changes to take effect. You can do this in two different ways:
1. Select Upgrade All Clusters errand when applying the changes in Ops Manager.
2. Manually run "tkgi upgrade-cluster <cluster-name>" for each cluster after applying the configuration, without using the Upgrade All Clusters errand.

Option B: Tune GC Thresholds via Kubernetes Profiles

Alternatively, leave the eviction thresholds alone and force Garbage Collection to happen earlier using a Kubernetes Profile.

Create a Kubernetes Profile JSON file that targets the kubelet component.

Define the exact GC thresholds required. For example, to start garbage collection at 80% disk usage and stop at 75%:

{
    "name": "kubelet-gc-tuning",
    "description": "Custom Kubelet GC Thresholds",
    "experimental_customizations": {
        "kubelet": {
            "imageGCHighThresholdPercent": 80,
            "imageGCLowThresholdPercent": 75
        }
    }
}

Apply the profile to the cluster via the CLI:
tkgi update-cluster cluster-#### --kubernetes-profile kubelet-gc-tuning

Note: Option A suits better if you want to the thresholds for all clusters in a Plan. If you want to change the thresholds on a an individual cluster, use Option B with Kubernetes Profile.