Log collection guidelines for Calico Node pod failure in TCA-deployed TKG workload cluster

Products

• VMware Telco Cloud Automation

Issue/Introduction

• This KB outlines the mandatory data collection steps required for Root Cause Analysis (RCA) of Calico CNI Node failures.
• Note: Collect these logs immediately after the issue occurs to prevent critical evidence from being lost due to log rotation.

Environment

TCA 2.x
TCA 3.x
TCP5

Resolution

Note: Collect logs immediately after the issue to avoid log rotation.Mention the approximate date/time of the occurrence of the issue 

For Root Cause Analysis (RCA), open a Broadcom Support case and attach the logs listed below.

1. Log the output of terminal into a file named commands_ouput.log.
   
   
2. Upload this log file with the case after running below mentioned commands.
   Note:If using putty select printable output only for logging option
   
   
3. Command Outputs from the Cluster with failed Calico Node Pod:
   
   

   # 1. Cluster State & Node Status
   kubectl get nodes -A -o wide
   kubectl get pods -A -o wide
   
   # 2. Identify the Problematic Pods
   # Replace <pod-name-fragment> with 'calico' or the specific pod ID
   kubectl get pods -A -o wide | grep <problematic_pod_name>
   
   # 3. Pod Details & Configuration
   # Replace <namespace> typically with 'kube-system', 'calico-system', or 'tkg-system'
   kubectl describe pod <problematic_calico_node_pod_name> -n <namespace>
   kubectl get pod <problematic_calico_node_pod_name> -n <namespace> -o yaml
   kubectl get pod <calico_controller_pod_name> -n <namespace> -o yaml
   
   # 4. Pod Logs (Current and Previous Instances)
   kubectl logs <problematic_calico_node_pod_name> -n <namespace> --previous
   kubectl logs <problematic_calico_node_pod_name> -n <namespace>
   kubectl logs <calico_controller_pod_name> -n <namespace>
   
   # 5. Events & System Health
   kubectl get events -A --sort-by='.lastTimestamp'
   kubectl top pod -n <namespace>
   kubectl -n kube-system logs ds/kube-proxy

   
   
   
4. SSH into the specific Worker/Control Plane Node VM where the failed calico-node pod was scheduled:
   
   

   #1. Container Runtime & Process Checks
   # Switch to root
   sudo -i
   
   # Basic Info
   uname -a
   date
   uptime
   
   # Container Runtime Inspection (CRI)
   crictl ps -a
   crictl pods
   crictl ps -a | grep <problematic_calico_pod_ID_or_name>
   # Inspect the failed container details
   crictl ps -a | grep <calico_pod_name> | awk '{print $1}' | xargs -r crictl inspect
   
   # Process List & Resource Usage
   ps -ef
   ps aux --sort=-%mem | head -10
   top -b -n 1
   free -h
   cat /proc/meminfo
   
   # Containerd Specifics (Check for 'gwpmm' or specific Calico processes)
   ctr -n k8s.io containers list | grep calico
   ctr -n k8s.io containers list | grep gwpmm
   
   #2. OS & Kernel Logs
   # Kernel Ring Buffer
   dmesg -T
   
   # System Journals
   journalctl --no-pager
   journalctl -u kubelet --no-pager
   journalctl -u containerd --no-pager
   
   # Collect last 48 hours of general logs
   journalctl --since "48 hours ago" --no-pager
   
   #3. Interface & Routing
   ip addr show
   ip route show
   ip neighbor show
   netstat -tulnp | grep -E '179|9099'  # Check BGP (179) and Felix (9099) ports
   
   # CNI Configuration Files
   ls -la /etc/cni/net.d/
   cat /etc/cni/net.d/*
   
   # IPTables & IPSets (Calico relies heavily on these)
   iptables-save > /tmp/iptables_dump.log
   ipset list > /tmp/ipset_dump.log
   
   #4. File System Checks
   df -hT
   df -i
   

    

5. Login to the Node where the failed calico node Pod was scheduled and collect below files:
   
   tar -czvf /tmp/var-log-backup-<Replace-with-actual--name-of-the-Node/VM>-$(date +%Y%m%d).tar.gz /var/log


6. Login to the control plane Nodes and collect files as in above step:
   
   tar -czvf /tmp/var-log-backup-<Replace-with-actual--name-of-the-Node/VM>-$(date +%Y%m%d).tar.gz /var/log


7. Collect the vCenter and relevant ESXI hosts support bundles where the control plane and worker node VMs with problematic pod were running.
   
   
8. Collect the TCA-M and TCA-CP support bundles, ensuring DB Dump and Kubernetes Logs are also selected.
   
   
9. Upload this information with the RCA case.