Increase ephemeral storage on vSphere Kubernetes Service (VKS) nodes

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• vSphere Kubernetes Service (VKS) nodes utilize an immutable OVA template with a fixed 20 GiB root partition.
• This partition hosts the OS, container images, and pod ephemeral data. When workloads require extensive storage for container images or large temporary files, the root filesystem may reach capacity, leading to cluster instability.
• Users may encounter the following issues:
  • Nodes report 'DiskPressure' status.
  • Pods are evicted due to insufficient ephemeral storage.
  • Large container images or log files consume the available space on the root (/) filesystem.

Environment

• vSphere Kubernetes Service
• Tanzu Kubernetes Grid (TKG) Service

Cause

The VKS node images are built with a static 20 GiB disk for the root partition. Directly resizing this partition is not supported as it is part of the immutable node image.

Resolution

To increase the storage capacity available to a node, add Node Volume Mounts to the cluster manifest. This attaches additional vSphere storage to specific directories like /var/lib/containerd (for images) or /var/lib/kubelet (for pod storage), offloading them from the root partition.

Configure Node Volume Mounts

1. Identify the vSphere Storage Policy to use.
2. Edit the Cluster manifest:
   kubectl edit cluster <cluster-name>
   
   
3. Add the volume configuration under the worker node pool variables.
   Example Configuration (v1beta1 API):
   
   variables:
  - name: nodePoolVolumes
    value:
      - name: containerd
        capacity:
          storage: 50Gi
        mountPath: /var/lib/containerd
        storageClass: <storage-class-name>
      - name: kubelet
        capacity:
          storage: 50Gi
        mountPath: /var/lib/kubelet
        storageClass: <storage-class-name>
4. Save the changes and exit the editor.
5. The Tanzu Cluster Operator performs a rolling update to replace existing nodes with new nodes containing the additional volumes.

Additional Information

Ephemeral storage uses the disk available on a node (space allocated to /).
The recommended and supported action is adding a short-lived Persistent Volume to allow additional storage to be consumed by pods on the node. The underlying volume is not stored on the node's disk, which you might expect with ephemeral storage volume. The volume is stored on the storage device you allocate to the underlying storage class and storage policy.

Using Storage Classes for Persistent Volumes

Create a vSphere Storage Policy for VKS Clusters

To get assistance from a customer representative or a Support Engineer, check Contact Broadcom Support