RKE2 / Canal CNI initialization failure caused by a host security agent reverting kube-proxy's firewall rules

Products

• Network Observability

Issue/Introduction

Canal's install-cni container loops indefinitely and never completes, with the pod stuck in Init:CrashLoopBackOff. The error is:

[FATAL] Unable to create token for CNI kubeconfig

  error=Post "https://:443/api/v1/namespaces/kube-system/serviceaccounts/canal/token":

  dial tcp :443: i/o timeout

The node may still report Ready, but any pod depending on the CNI (CoreDNS, ingress, metrics-server, etc.) stays Pending/ContainerCreating, and /etc/cni/net.d stays empty because install-cni fatals before writing the final config. A full reboot does not resolve it.

Environment

• RKE2 with Canal (Calico + Flannel) CNI.
• RHEL 8.x or 9.x control-plane node.
• A host-based security/firewall-management agent installed alongside RKE2 — confirmed with Illumio VEN; also check for Tanium and Trellix/McAfee ENS on the same host.
• Reproduces on fresh installs and upgrades alike — it's an environment condition, not tied to a specific product version.

Cause

The Kubernetes service ClusterIP isn't a real listener — it only works because kube-proxy writes NAT rules that redirect it to the real API server address. The security agent detects those rule writes as unauthorized changes to the host firewall and reverts them, deleting kube-proxy's chains faster than it can keep them in place. install-cni calls the service ClusterIP directly (by design, so it doesn't depend on the pod network being up yet), so when the redirect rule is gone, the call times out and the container fatals.

This is not an RKE2, Canal, or installer defect. It doesn't self-heal on reboot because the agent's revert behavior re-arms at boot and then runs continuously from there.

Confirmation (must run as root):

• A request to the local API server endpoint succeeds, but a request to the Kubernetes service ClusterIP times out — isolates the failure to the service-routing path, not the API server itself.
• kube-proxy's log shows iptables-restore ... Chain 'KUBE-SVC-...' does not exist.
• Decisive: the agent's own log admits it —
• grep -i tamper | tail -50

Repeated FW tampering has been detected / Successfully reverted out of band firewall changes entries, every few seconds to tens of seconds.

Resolution

Do not disable the agent, pin iptables-legacy, hand-edit CNI files, or repeatedly flush iptables — none of that addresses the cause and the flushing makes it worse.

Get the agent's administrators (console/PCE-side, not just the local host) to add a policy exception covering the pod network range, the service network range (including the Kubernetes service ClusterIP), and the CNI overlay traffic (e.g. flannel VXLAN). Specifically ask them to check for a tamper-protection/firewall-coexistence setting — separate from the general traffic-enforcement mode, which may already be permissive — and exempt kube-proxy's chains (KUBE-SERVICES, KUBE-SVC-*, KUBE-SEP-*) from it. Once the exception is applied, delete the crash-looping Canal pod — no reinstall or iptables flush required.