Root Partition Full: Log Rotation Failure for Antrea NSX Interworking Adapter on VKS Nodes

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

^{Administrators may observe high root disk usage or "disk pressure" alarms on worker nodes within a vSphere Kubernetes Service (VKS) cluster. This occurs specifically when the cluster is integrated with the Antrea NSX Adapter (interworking adapter).}

^Symptoms:

• ^{VKS worker nodes report high utilization of the root (/) partition.}
• ^{Large log files are discovered in the /var/log/interworking}
• ^{On a problematic node, the following command shows log files exceeding the expected 25 MB threshold: ls -lh /var/log/interworking}

Environment

• ^{Platform: vSphere with Tanzu / VKS}
• ^{Component: Antrea NSX Interworking Adapter (antrea-interworking pod)}
• ^{Versions: Impacted versions prior to vkr 1.33}

Cause

^{By default, the Antrea Interworking Adapter is configured with a maximum log file size of 25 MB. However, a logic defect prevents the log rotation mechanism from triggering when the antrea-interworking pod runs on a VKS node for the first time without any restarts.}

^{If the container remains in a continuous "Running" state from its initial deployment, the logs will continue to grow indefinitely, consuming available disk space on the host node. Notably, if the container restarts at least once for any reason, the rotation logic initializes correctly, and the issue does not persist.}

Resolution

^{Fixed Versions}

^{This issue is officially resolved in the following releases:}

• ^{vkr 1.33, 1.34, 1.35 and later.}

^{Broadcom recommends upgrading to these versions to resolve the underlying rotation logic defect.}

^Workaround

^{If an immediate upgrade is not possible, implement the following two-step workaround to manage disk usage effectively.}

^{Step 1: Immediate Truncation (Manual)}

^{To reclaim disk space immediately on a node currently experiencing high usage, truncate the large log file:}

^{# Identify the large log file and truncate it echo > /var/log/interworking/<large_log_file_name>}

^{Step 2: Automated Log Rotation (Deployment)}

^{Apply a logrotate configuration to the cluster to manage these files automatically.}

1. ^{Download the yml file attached to this article.}
2. ^{Apply the configuration to the VKS cluster running the antrea-interworking adapter: kubectl apply -f logrotate.yml}
3. ^{This configuration will monitor the /var/log/interworking directory and rotate files once they reach the maximum size defined in the deployment.}

Additional Information

^{For more details on VKS and Antrea CNI integration, please refer to the official Broadcom documentation:}

• ^{Securing VKS: Solution Components - VKS Clusters Antrea CNI Integration}