Pods in OpenShift Cluster Stuck in ContainerCreating State
Article ID: 423913
Updated On:
Products
Issue/Introduction
- In an OpenShift environment integrated with VMware NSX, Pods may fail to start and remain indefinitely in the
ContainerCreatingstate. - This issue occurs when both containers inside the nsx-node-agent Pod (nsx-node-agent and nsx-kube-proxy) attempt to communicate with the Kubernetes API server using an expired ServiceAccount token.
- For the NSX node agent pod, the nsx-node-agent and/or the nsx-kube-proxy container report the following error while handling a CNI ADD request:
2025-12-25T06:29:59.344Z ########-########-########-######## NSX 17 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="WARNING"] nsx_ujo.agent.cni_watcher Got an exception when processing CNI request, Failed pods request: Failed to get pods : ########-statefulset-0, error: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
- This issue typically affects nodes where the nsx-node-agent Pods have been running continuously for more than 365 days without a restart.
- x509 error might also be seen
"Error from server: Get "https://#.#.#.#:10250/containerLogs/openshift-apiserver/apiserver-##########-####/openshift-apiserver": tls: failed to verify certificate: x509: certificate signed by unknown authority"
- Creation of new containers might failed with below error events.
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 27s (x12414 over 47h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_apiserver-##########-####_openshift-apiserver_########-####-####-####-############_#(#####################################################################): error adding pod openshift-apiserver_apiserver-##########-#### to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400:
>>>{ContainerID Details}<<<
Path:"" ERRORED: error configuring pod [openshift-apiserver/apiserver-##########-####] networking: [openshift-apiserver/apiserver-##########-####/########-####-####-####-############:nsx-cni]: error adding container to network "nsx-cni": plugin type="nsx" failed (add): Failed to receive message header:EOF.
Environment
VMware NSX
NSX Container Plugin
OpenShift Container
Cause
The nsx-node-agent and nsx-kube-proxy containers read the Kubernetes ServiceAccount token only at startup and keep using the same HTTP session to communicate with the Kubernetes API server. Since these containers had been running continuously for more than 365 days without a restart, the ServiceAccount token expired, causing all Kubernetes API requests to fail with a 401 Unauthorized error and preventing Pods from being created.
Resolution
Workaround
Perform a rollout restart of the nsx-node-agent DaemonSet to refresh the ServiceAccount token.
Feedback
