Pods Stuck in Init State due to Istio-CNI iptables Duplicate Chain Error

Products

• VMware Telco Cloud Automation

Issue/Introduction

 In VMware Telco Cloud Automation (TCA) environments, application pods may spontaneously enter and remain in a persistent Init state following a Node Kernel Version Upgrade. The primary symptom is a failure in the istio-cni network configuration, characterized by the error: error cni Command error output: xtables other problem: line 2 failed: Chain already exists. This prevents the pod from initializing its networking and progressing to a "Running" state unless the worker node is rebooted.

Environment

VMware Telco Cloud Automation 3.2

Cause

The issue is caused by a defect in the Istio-CNI iptables detection logic. In certain environments, iptables-restore is aliased to iptables-nft-restore.

During the pod initialization sequence, Istio-CNI incorrectly triggers both iptables-restore --noflush and iptables-nft-restore --noflush. Because these commands are effectively identical in the underlying system, the networking rules are applied twice. This leads to a conflict where the second command fails because the required iptables chains already exist. Additionally, log reviews may show a segfault in libnftnl.so.11.2.0 related to iptables-nft-sa during this process.

Resolution

To permanently resolve this issue, upgrade Istio to a version that includes improved iptables binary detection logic.

1. Validate Upgrade Path: Ensure your current TCA and TKG versions support the target Istio version.
2. Upgrade Istio: Upgrade to Istio version 1.23 or higher.
   • Versions 1.23+ and 1.25+ contain specific enhancements to how the CNI detects and handles nft vs legacy iptables binaries, preventing duplicate execution.
3. Verification:
   • After the upgrade, monitor the pod startup sequence using:
     

     kubectl get pods -A -o wide | grep <pod_name>

   • Verify the istio-cni-node logs to ensure only one set of iptables restore commands is being executed without "Chain already exists" errors.

Workaround If an immediate upgrade is not possible, the following temporary mitigation can be used:

• Node Reboot: A fresh reboot of the affected Worker Node typically clears the inconsistent state and allows pods to initialize correctly once, though the issue may recur during subsequent kernel updates or network resets.