pod running on containerd runtime in unprivileged mode getting error bind port permission denied

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Some pods that were initially running successfully on Docker runtime (pod is able to bind to any port <1024) may unexpectedly crash after the migration from Docker to ContainerD runtime.

After the migration to ContainerD runtime, those pods may go into crashloopbackoff with a permission denied error when binding port operation is performed, a snippet from the pod logs kubectl logs <PODNAME>

bind() to 0.0.0.0:80 failed (13: Permission denied)

 

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Resolution

Due to the migration from Docker and ContainerD runtime, some security features are handled in different way.

nginx Docker project has changed port 80 to a port greater than 1024 to solve this permission issue https://hub.docker.com/r/nginxinc/nginx-unprivileged

If the developer cannot change the default port to a port greater than 1024, the following security context can be added (example for a pod experiencing the permission issue on port 80:

      securityContext:
        sysctls:
        - name: net.ipv4.ip_unprivileged_port_start
          value: "80"

Example of nginx pod running in unprivileged user and bind to port 80

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      securityContext:
        sysctls:
        - name: net.ipv4.ip_unprivileged_port_start
          value: "80"
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.16.1-alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-conf

The pod should now be able to start and bind to port 80 using ContainerD runtime.
Similar approach can be completed with different low range level port (<1024) like: 433 53 22 ...

 

 

Additional Information

Further info about the securityContext ip_unprivileged_port_start in Kubernetes documentation https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#enabling-unsafe-sysctls and Kernel documentation https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt . The default ip_unprivileged_port_start is 1024, same default value is set in Xenial stemcell.

Kubernetes upstream issue https://github.com/kubernetes/kubernetes/issues/56374