Error: "The node was low on resource: ephemeral-storage"

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Pods in evicted state
• Error message in kubectl describe pod ####:

The node was low on resource: ephemeral-storage. Threshold quantity: #####, available: ####Ki. Container web-ui-image was using ###Ki, request is 0, has larger consumption of ephemeral-storage. 

• Disk pressure warning on nodepool

Environment

3.2

Cause

• Container(s) using excess node (local) storage causing the pods to get evicted and be unable to start.

Resolution

Identify container/plugin utilizing the ephemeral storage

1. Run the following command to identify directories with large storage utilization
   du -sh ./*
2. If the results return the /var/lib/kubelet/plugins directory, consult the container/plugin documentation to ensure directory can be cleared without a negative impact on the container/plugin.

Address excess ephemeral storage usage

• Address ephemeral storage usage via the yaml file.
• Ephemeral volume types emptyDir, configMap, downwardAPI, and secret are provided as local ephemeral storage. They are managed by kubelet on each node. See Ephemeral Volumes for more info.