DX AIOps - pods in CrashLoopBackOff , error "OCI runtime create failed: runc create failed: unable to start container process"

Products

• DX Operational Intelligence
• DX Application Performance Management

Issue/Introduction

Some pods remain in CrashLoopBackOff

Example1:

kubectl get pods -n <your-dxi-namespace> | grep CrashLoopBackOff    

doi-metric-metadata-<pod-id>                   0/1     CrashLoopBackOff    3         
doireadserver-<pod-id>                          0/1     CrashLoopBackOff    2      
dxi-notify-<pod-id>                          0/1     CrashLoopBackOff    2      

kubectl get events -n <your-dxi-namespace> | grep Error 

Warning  Failed     115s (x4 over 2m38s)  kubelet            Error: failed to start container "doireadserver": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "entrypoint.sh": cannot run executable found relative to current directory: unknown

 

Example 2:

kubectl get pods -n <your-dxi-namespace> | grep CrashLoopBackOff    

      doi-normalized-alarm-<your-dxi-namespace>                  0/1     CrashLoopBackOff   25         

kubectl get events -n <your-dxi-namespace> | grep Error 

Error: failed to start container "normalized-alarm": Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "entrypoint.sh": cannot run executable found relative to current directory: unknown

 

Environment

DX Platform 21.3

Resolution

1) Edit the problematic deployment(s)

 

kubectl edit deployment <affected deployment> -n namespace

 

3) Add the full path of the entrypoint.sh as below

 

-command:

 - /opt/entrypoint.sh
 env:

 

For example

 

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html