Pods become evicted due to DiskPressure issues

Products

Tanzu Mission Control VMware Tanzu Kubernetes Grid Service (TKGs) VMware Tanzu Kubernetes Grid Management VMware Tanzu Kubernetes Grid Integrated (TKGi) VMware Integrated OpenStack

Issue/Introduction

Pods become "Evicted" due to DiskPressure.
This might occur on ControlPlane or Worker Nodes.

Environment

VMware Integrated Openstack 7.x
VMware Tanzu Kubernetes Grid Integrated Edition 1.20.0/1.21.0
Tanzu Mission Control 1.4.0/1.3.1

Cause

This issue is presented by upstream Kubernetes constraints, it has potential to impact any product and version that uses Kubernetes orchestration.

K8s will evict pods when available disk space and inodes on either the node's root filesystem or image filesystem (likely disk space in /var/lib/docker, /var/tmp/journal/ or /var/log) or container filesystem has satisfied an eviction threshold. See Kubernetes documentation on Node-Pressure Eviction for details.
Additional errors lead to the solution:

init container start failed: ErrImageNeverPull: Container image "vmware/vio/utils/data-aggregator:v0.3_photon3" is not present with pull policy of Never"

Resolution

Full resolution for this will require investigation into the application workloads and images running on the impacted nodes to see where they are consuming node disk space and how much is consumed. If the application is using images that are larger than the imagefs can accommodate, the node's disk size will need to be increased. If increasing the node's disk size is not an option, review the workaround below

Workaround:

Clean up of the evicted pods can be done with the following command:

kubectl get pods --all-namespaces | grep Evicted | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod

Depending on the number of evicted pods, this could take a few minutes to complete. If the command hangs, you can ctrl-c and re-run the command.

OR

Identify directory and large log files on vio manager:

df -h

find / -xdev -size +10M -print | xargs ls -lS

Once identified, you can clean out log files or older files from pods that may have already been removed especially if they are larger than 2GB, either by deleting or moving them to a larger storage unused by vio.

Additional Information

For Vmware Integrated OpenStack:

See this kb for freeing up space on controller nodes. Patch failed with errors 'node has conditions: [DiskPressure]'

In addition to eviction of pods, we could also face a situation where docker images are deleted as well. There are two potential scenarios for docker images to be missing.

Free disk space was lower than 15% at one point. This could be caused by uploading a large image using glance or horizon. kubelet will delete containers and images to free some space.
Manually running 'docker system prune -a' commmand. This command will delete images which are not used by any running container. ie calico-kube-controller pod is not running on this k8s node, vmware/calico/kube-controllers:v3.8.2 will be deleted. This should not be done unless directed by support.

Bootstrap images (image name starts with "vmware/" instead of "docker-registry.default.svc.cluster.local:5000/") can not be recovered automatically if deleted. They can only be recovered by loading the tar file from vio-manager.

Copy /opt/vmware/data/bootstrap-images/bootstrap-images.tar from vio-manager to this controller and run 'docker load -i'

Example:
scp -i ~/.ssh/vsphere_tmp /opt/vmware/data/bootstrap-images/bootstrap-images.tar vioadmin@controller-##########:/tmp/
viossh controller-##########
docker load -i /tmp/bootstrap-images.tar

NOTE: If running on VIO Manager itself you can run the following command:

docker load -i /opt/vmware/data/bootstrap-images/bootstrap-images.tar

For vSphere with Tanzu clusters:

Reference the Support for Node Volume Mounts for TKG Clusters in Supervisor which details supportability and best practices for node volume mounts to increase the default node disk size.