POD not starting due to system file limit

Products

• DX Operational Observability
• DX Operational Intelligence

Issue/Introduction

We are unable to start the POD with the system limit error below, how can we fix it?

UNABLE TO START, System file handles limit is less than required(65536) : 1024       

Environment

DX O2 25x and 26x on premise.

Cause

The nofile ulimit must be properly set in two layers:

• Worker node OS (fs.file-max): already correct (> 65536)
• Container runtime (LimitNOFILE): defaulting to 1024 — this is the fix needed

The container runtime (containerd or CRI-O) has its own default nofile limit that overrides the OS setting unless explicitly configured.

Resolution

Step 1 — Verify the current ulimit on the worker node

SSH into each worker node where the pod can be scheduled and run:

ulimit -n
cat /proc/sys/fs/file-max

Confirm the OS-level value is already above 65536. If not, proceed to Step 2 before continuing.

Step 2 — Set the OS-level limits (if not already done)

Edit /etc/security/limits.conf and add:
* soft nofile 65536
* hard nofile 65536
root soft nofile 65536
root hard nofile 65536

Edit /etc/sysctl.conf and add or update:
fs.file-max = 524288

Apply the sysctl change immediately:
sysctl -p

Step 3 — Configure the container runtime

For containerd, create or edit the override file:
mkdir -p /etc/systemd/system/containerd.service.d/
vi /etc/systemd/system/containerd.service.d/override.conf

Add the following content:
[Service]
LimitNOFILE=65536

For CRI-O, create or edit:
mkdir -p /etc/systemd/system/crio.service.d/
vi /etc/systemd/system/crio.service.d/override.conf

Add the same content:
[Service]
LimitNOFILE=65536

Step 4 — Reload systemd and restart the container runtime

Restarting the container runtime will briefly interrupt all running pods on that node. Plan this during a maintenance window or cordon the node first.

To safely cordon and drain the node before restarting:
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Restart the container runtime:

systemctl daemon-reload
systemctl restart containerd   # or: systemctl restart crio

After restarting, uncordon the node:

kubectl uncordon <node-name>

Step 5 — Repeat on all affected worker nodes

Apply Steps 2 through 4 on every worker node where the logs-archival-default pod can be scheduled.

Step 6 — Delete the pod to force a reschedule

kubectl delete pod logs-archival-default-0 -n <namespace>

Step 7 — Verify the fix inside the pod

Once the pod is running, confirm the ulimit is correct inside the container:
kubectl exec -it logs-archival-default-0 -n <namespace> -- sh -c "ulimit -n"

Expected output: 65536 or higher. If confirmed, the service should start normally.