Pod calico-node is not in Ready State (0/1) and pod calico-kube-controllers is in CrashLoopBackOff state on TCA Upgrade

Products

• VMware Telco Cloud Automation

Issue/Introduction

On TCA Upgrade after VM Reboot, the pod calico-node is not in Ready State (0/1) and the pods calico-kube-controllers is in CrashLoopBackOff state.

root@dual-tca-cp2 [ /home/admin ]# kubectl get pods -A|grep calico
NAMESPACE            NAME                                       READY   STATUS             RESTARTS         AGE
kube-system          calico-kube-controllers-857b857f4c-jdwfl   0/1     CrashLoopBackOff   130 (42s ago)    6h46m
kube-system          calico-node-bfwkp                          0/1     Running            0                6h46m

Logs of calico-node is reporting error with logs below
YYYY-MM-DD HH:MM:SS [PANIC][xxxxxx] felix/table.go 815: iptables-nft-save command failed after retries ipVersion=0x4 table="filter".

Environment

VMware Telco Cloud Automation 3.4

Cause

Calico is not able to execute command iptables-nft-save successfully because of a incompatible rule added by an external systems.

Resolution

Restart the Appliance VM as it auto triggers the upgrade. Node restart will flush the iptables rules and restore on startup without the incompatible rule. Ensure the calico-node and calico-kube-controller pods in Running state. 

Additional Information

Please see article below:

https://techdocs.broadcom.com/us/en/vmware-sde/telco-cloud/vmware-telco-cloud-automation/3-4/vmware-telco-cloud-automation-release-notes/vmware-telco-cloud-automation-34-release-notes.html