Networkpolicy and ALB routing is failing

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

When applying a network policy in order to allow only specific traffic to the DNS and to the ingress/httpproxy IP. the operation fails Sample of the policy applied for simplicity port 80 is allowed for all ips: 

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: network-policy-test
  namespace: network-policy-test
spec:
  egress:
  - ports:
    - port: 53
      protocol: UDP
    to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
  - ports:
    - port: 80
      protocol: TCP
    to:
    - ipBlock:
        cidr: 0.0.0.0/0
  policyTypes:
  - Egress

Having the following configuration in a lab without network policy:

kubectl get po,svc,httpproxy,networkpolicy -n network-policy-test
NAME                                         READY   STATUS    RESTARTS   AGE
pod/curl                                     1/1     Running   0          29m
pod/nginx-ping-deployment-84d9477d56-972ct   1/1     Running   0          45m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/nginx-ping   ClusterIP   100.67.xxx.82   <none>        80/TCP    45m

NAME                                        FQDN                TLS SECRET   STATUS   STATUS DESCRIPTION
httpproxy.projectcontour.io/nginx-ingress   test-alb.test.com                valid    Valid HTTPProxy

Curl command works from the curl pod to the httpproxy FQDN

curl -v  --resolve test-alb.test.com:80:10.159.xxx.116 http://test-alb.test.com
* Added test-alb.test.com:80:10.159.xxx.116 to DNS cache
* Hostname test-alb.test.com was found in DNS cache
*   Trying 10.159.xxx.116:80...
* Connected to test-alb.test.com (10.159.xxx.116) port 80 (#0)

When above network policy is applied even that port 80 is allowed there is timeout with same command

connect to 10.159.xxx.116 port 80 failed: Operation timed out

Environment

TKGm 2.5.x

ALB

CNI Antea

Ingress Contour/Envoy 

Cause

The reason for this to happen is rooted in the process of NATing happening when the pod is trying to access the FQDN.

pod resolves the address to the loadbalancer IP 10.159.xxx.116 and packet is directed to this IP.

However there is a firewall rule in iptables that is translating the address to a different port:

iptables-save | grep -E "10.159.xxx.116|KUBE-EXT-O7FQBTCCPLX4GWHO"
:KUBE-EXT-O7FQBTCCPLX4GWHO - [0:0]
-A KUBE-EXT-O7FQBTCCPLX4GWHO -m comment --comment "masquerade traffic for tanzu-system-ingress/envoy:http external destinations" -j KUBE-MARK-MASQ
-A KUBE-EXT-O7FQBTCCPLX4GWHO -j KUBE-SVC-O7FQBTCCPLX4GWHO
-A KUBE-NODEPORTS -p tcp -m comment --comment "tanzu-system-ingress/envoy:http" -m tcp --dport 30442 -j KUBE-EXT-O7FQBTCCPLX4GWHO
-A KUBE-SERVICES -d 10.159.xxx.116/32 -p tcp -m comment --comment "tanzu-system-ingress/envoy:https loadbalancer IP" -m tcp --dport 443 -j KUBE-EXT-U74SBURFS3P27VO4
-A KUBE-SERVICES -d 10.159.xxx.116/32 -p tcp -m comment --comment "tanzu-system-ingress/envoy:http loadbalancer IP" -m tcp --dport 80 -j KUBE-EXT-O7FQBTCCPLX4GWHO 

 KUBE-EXT-O7FQBTCCPLX4GWHO  - is the rule that completes the translation to nodeport 30442  corresponding to the envoy service where it redirects the request to a envoy pod IP and port 8080

-A KUBE-SVC-O7FQBTCCPLX4GWHO -m comment --comment "tanzu-system-ingress/envoy:http -> 100.xx.1.8:8080" -j KUBE-SEP-MGPQIHNL5WITKL3H

However based on our rule - the port 8080 is not in the allowed list and therefore the request is dropped.

Another method to confirm this problem is using antreactl with below procedure:

Exec into antrea controller in kube-system namespace:

kubectl exec -it -n kube-system antrea-controller-xxxxxxxx-xxxx -- bash

Execute traceflow to capture the traffic in question:

antctl traceflow -S network-policy-test/curl -D 10.159.xxx.116 -f tcp --live-traffic --dropped-only -t 1m

Where:

-S network-policy-test/curl - source is from namespacenetwork-policy-test and curl pod

 -D 10.159.xxx.116 - is the destination IP 

-f tcp --live-traffic --dropped-only -t 1m - capture tcp only dropped packets in life stream and run with 1 minute timeout

execute from the test curl pod the above curl command the expected output is provided below:

name: network-policy-test-curl-to-10.159.xxx.116-d8xs42vn
phase: Succeeded
source: network-policy-test/curl
destination: 10.159.xxx.116
results:
- node: workload-slot1rp4-md-0-64d5cf6d58xfq69w-gclt7
  timestamp: 1734704920
  observations:
  - component: SpoofGuard
    action: Forwarded
  - component: LB
    action: Forwarded
    translatedDstIP: 100.xx.1.8
  - component: NetworkPolicy
    componentInfo: EgressDefaultRule
    action: Dropped
capturedPacket:
  srcIP: 100.xx.1.37
  dstIP: 100.xx.1.8
  length: 60
  ipHeader:
    protocol: 6
    ttl: 64
    flags: 2
  tranportHeader:
    tcp:
      srcport: 45516
      dstport: 8080
      flags: 2

As visible from the traceflow packet was dropped after it was forwarded to the envoy on different port 

 

Resolution

Updating the network policy to allow port 8080 to the envoy pod is at very minimum to allow the operation to be completed 

  - ports:
    - port: 8080
      protocol: TCP
    to:
    - ipBlock:
        cidr: 100.xx.1.8/32  # Allow traffic to the translated ClusterIP