Pods stuck in CrashLoopBackOff state

Products

• VMware Telco Cloud Automation

Issue/Introduction

The application pods are stuck in a CrashLoopBackOff state. If you describe the pod and check the events, you see a warning similar to the one below:

Warning  FailedCreatePodSandBox  2m32s (x41 over 3m48s)  kubelet                  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "UUID": plugin type="multus" name="multus-cni-network" failed (add): [NAMESPACE/POD]: error adding container to network "NETWORK": netplugin failed: "2024-08-22T16:44:17Z [debug] Used defaults from parsed flat file config @ /etc/cni/net.d/whereabouts.d/whereabouts.conf\n2024-08-22T16:44:17Z [debug] ADD - IPAM configuration successfully read: {Name:POD Type:whereabouts Routes:[{Dst:{IP:DEST-IP Mask:ffffffff} GW:GATEWAY} {Dst:{IP:DEST-IP Mask:ffffffff} GW:GATEWAY} {Dst:{IP:DEST-IP Mask:ffffffff} GW:GATEWAY}] Datastore:DATASTORE Addresses:[] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range:CIDR RangeStart:CIDR-START RangeEnd:CIDR-END GatewayStr: EtcdHost: EtcdUsername: EtcdPassword:********* EtcdKeyFile: EtcdCertFile: EtcdCACertFile: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile: LogLevel: OverlappingRanges:true SleepForRace:0 Gateway:<nil> Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath: PodName:POD PodNamespace:NAMESPACE}\n2024-08-22T16:44:17Z [debug] Beginning IPAM for ContainerID: UUID\nruntime: failed to create new OS thread (have 14 already; errno=11)\nruntime: may need to increase max user processes (ulimit -u)\nfatal error: newosproc\nruntime: failed to create new OS thread (have 14 already; errno=11)\nruntime: may need to increase max user processes (ulimit -u)\nfatal error: newosproc\n\nruntime stack:\nruntime.throw(0x17a6fb7, 0x9)\n\t/usr/local/go/src/runtime/panic.go:1116

 

Environment

2.x

Cause

This is due to the application being configured with too low a value for kernel.pid.max in the CSAR.

Resolution

Increasing the kernel.pid.max value will resolve the issue. The necessary value will need to be determined by the application vendor.