File Descriptor limit in TKG Nodes

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• A discrepancy in File Descriptor (FD) limits is observed between different environments (e.g. Production vs. Staging).
• Observed Configuration:

  • Production Environment:

    • Soft Limit: 1024

    • Hard Limit: 4096

  • Staging Environment:

    • Soft Limit: 96000

    • Hard Limit: 96000

• Clarity is required regarding the system impact of increasing the Production limits to match the staging environment configuration.
• This article provides an analysis of the resource impact (CPU/Memory) when increasing File Descriptor limits on Tanzu Kubernetes nodes and differentiates between shell-level limits (ulimit) and system-wide limits (fs.file-max).

Environment

TCA 2.3
Kubernetes 1.24.10

Resolution

Interactive Shell Limits vs. System Limits

The values typically returned by the ulimit -n or ulimit -Hn commands in an SSH session reflect the limits of that specific interactive shell session. These are controlled by PAM configuration files (e.g., /etc/security/limits.conf).

It is observed that Kubernetes Pods and Systemd services do not inherit limits from the interactive SSH shell. These services use limits defined in their respective systemd unit files (LimitNOFILE) or container runtime configurations. Therefore, the limit of "4096" observed in an SSH session may not represent the actual limit enforced on Kubernetes workloads.

System-Wide Limits (fs.file-max)

The Linux kernel enforces a global ceiling for file descriptors across the entire operating system, defined by the fs.file-max parameter. In supported Photon OS versions, this value is typically calculated based on available RAM or set to a high default value to prevent system-wide bottlenecks.

To verify the system-wide limit, the following command is used:

sysctl fs.file-max

Impact Assessment

Increasing the File Descriptor Hard Limit (e.g., from 4096 to 96000) results in the following resource implications:

1. Configuration Overhead
   Changing the configuration limit itself has no performance impact. The kernel does not pre-allocate resources based on the limit value; it serves only as a ceiling for requests.
2. Kernel Memory Impact
   When an application utilizes the increased limit to open additional files, the kernel must track them using the struct file object.
   • Memory Cost: Approximately 1 KB of non-swappable kernel memory is consumed per open file.
   • Estimated Impact: Opening 96,000 files would consume approximately 96 MB of kernel RAM (96,000 * 1 KB).
   • Conclusion: This overhead is considered negligible for modern server infrastructure.
3. Application Memory Impact
   The primary resource consumption occurs within the application itself when it utilizes the open connections.
   • Memory Cost: Applications allocate buffers (e.g., read/write buffers) for every open network socket.
   • Estimated Impact: If an application maintains 96,000 concurrent connections with a 64 KB buffer per connection, it would require approximately 6.1 GB of RAM.
   • Conclusion: Capacity planning is required to ensure Worker Nodes have sufficient RAM to support the workload concurrency that the higher limit permits. The limit increase itself is safe.
4. CPU Impact
   There is no direct CPU cost associated with a higher limit configuration. However, processing a higher volume of concurrent connections (e.g., 96,000) will result in increased CPU utilization due to context switching and packet processing.

Verification

To determine the specific limits applied to a running process or container (bypassing the SSH shell limits), perform the following steps:

1. Identify the Process ID (PID) of the application:

   pidof <process_name>
   

2. Inspect the limits for that specific PID:

   cat /proc/<PID>/limits
   

3. Locate the row labeled Max open files. This value represents the active limit enforced on the process.