Pods getting evicted because of low ephemeral-storage in VMware Enterprise PKS Environment

Products

• VMware Cloud PKS

Issue/Introduction

This article provides instructions for making a default directory for local volume mounts (like emptyDir) to the ephemeral disk (which maps to /var/vcap/data). 

Symptoms:

1. When you run following command , you will notice that pod status shows as 'evicted'.

kubectl get pod -n <namespace-name>

2. When you describe the evicted pod using following command, you will see a message stating that 'The node was low on resource: ephemeral-storage'

kubectl describe pod <pod-name> -n <namespace-name>

Cause

In Enterprise PKS 1.3.6 & below, the default directory for local volume mounts (like emptyDir) is /var/lib/kubelet, which maps to the root disk and is only 3GB in size.

Resolution

This is a known issue affecting Enterprise PKS 1.3.6 and below. This issue is resolved in Enterprise PKS 1.3.7 and above.

Workaround:
Use the following steps to workaround this issue:
 

1. Create an opsfile similar to the following to set the root-dir . 

cat /tmp/opsfile-set-kubelet-root-dir.yaml
- type: replace
  path: /instance_groups/name=worker?/jobs/name=kubelet/properties/k8s-args/root-dir
  value: ((root_dir))

2. Issue a command similar to the following to download the manifest of a target cluster via BOSH

bosh manifest -d service-instance_4d52f36a-8075-437d-b386-1b92d1322a28 > /tmp/my-manifest.yaml

3. Issue a command similar to the following to Interpolate the downloaded manifest from Step 2 with the opsfile from Step 1 and variable to make sure you see your desired output

bosh interpolate /tmp/my-manifest.yaml -o /tmp/opsfile-set-kubelet-root-dir.yaml  -v root_dir="/var/vcap/data/kubelet" | grep root-dir

         

Note: You should see output similar to the following:
 
root-dir: /var/vcap/data/kubelet

4. Issue a command similar to the following to apply the changes to the cluster:

bosh deploy -d service-instance_4d52f36a-8075-437d-b386-1b92d1322a28 /tmp/my-manifest.yaml -o /tmp/opsfile-set-kubelet-root-dir.yaml -v root_dir="/var/vcap/data/kubelet"
 

Notes:

1. The changes will be lost when you upgrade the PKS tile or do a resize on the cluster. Resurrection will keep the existing changes.
2. When you copy from the text editor to the cli, type the double quotes manually. This will prevent you from running into any issues with improper quotation marks being used.

Additional Information

In VMware Enterprise PKS 1.3.7 and above, the default directory for local volume mounts (like emptyDir) is /var/vcap/data, which maps to the ephemeral disk.