Error ""exec format error" observed during application Pod failure

Products

• VMware Telco Cloud Automation

Issue/Introduction

When deploying application Pods, they fail to start and enter a CrashLoopBackOff or Error state.

NAME                               READY            STATUS                   RESTARTS
app-data-plane-####                 0/1           CrashLoopBackOff             5

kubectl logs <pod-name> -n <name space> reveals the following error:

exec /usr/local/bin/entrypoint: exec format error

Environment

2.x,3.x

Cause

• The error exec format error occurs when the Linux Kernel on the worker node attempts to execute a binary file format that the CPU is unable to interpret.
• This occurs when there is a policy change to a node that has an effect on the CPU configuration. 

Resolution

Recreate Affected Worker Nodes

1. Identify the node where the affected pod is running.
   
   kubectl describe pod <pod name> -n <namespace> | grep "Node:"
   
   
2. Cordon the node to prevent new pod scheduling
   
   kubectl cordon <node name>
   
   
3. Drain the node to evict the existing pods
   
   kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data --force
   
   
4. Switch context to the Management Cluster:
   
   kubectl config use-context <mgmt-cluster-context>
   
   
5. Find the Machine object associated with the worker node:
   
   kubectl get machines -A | grep <node name>
   
   
6. Delete the Machine object:
   
   kubectl delete machine <machine-name> -n <namespace>
   
   
7. Monitor the node creation
   
   kubectl get nodes -w
   
   
8. Verify Pod Placement:
   
   kubectl get pods -n <pod name> -o wide