TKGm 2.x with AVI and package contour Envoy does not preserve Client IP

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

When TKGm is configured with AVI and contour package the following configuration is taking place:
When contour is created it is using a loadbalancer service which is realised as virtual service in AVI, once service and deployment is completed we have to configure httpproxy (ingress) so any of the pages registered to be forwarded to the respective K8s Application (cluster services) 
Some applications require to have information about the client who is reaching the endpoint, and for this reason they require to record the client IP address using http headers like "X-Forwarded-For"
Because AVI Virtual service is configured as a L4 loadbalancer there are two possible scenarios

In L4 mode, preserving client IP requires Avi to be configured with either proxy protocol or native client IP preservation, both of which require an Avi Enterprise license.

Since Avi is doing L4 only in this case, it can't set the X-Forwarded-For header as this option is only for HTTP L7 Virtual service, so that is not an option here.

Contour can be configured to operate using any of the above approaches:

• If using PROXY protocol, it needs to be enabled in Contour via the `contour.useProxyProtocol` package setting (ref. https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/2.3/using-tkg/workload-packages-contour.html#optional-configuration-5) 
• If using X-Forwarded-For (would require Avi to be doing L7), Contour needs to be configured with the correct number of trusted hops via `contour.configFileContents.network.num-trusted-hops` package setting (ref. https://projectcontour.io/docs/1.28/configuration/#network-configuration and https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/2.3/using-tkg/workload-packages-contour.html#optional-configuration-5)   This options is not available with the service of type L4

ref. https://docs.vmware.com/en/VMware-NSX-Advanced-Load-Balancer/22.1/Configuration_Guide/GUID-26DBF123-7DF2-4159-8198-CB979E41C19E.html 

If Contour is configured to use PROXY protocol then any packets that are not coming with PROXY protocol will be dropped so if a Pod within the cluster wants to access such service it will have to use PROXY protocol, which can be done via the flag curl --haproxy-protocol https://example.com/. 

You can find more information on the following linkhttps://everything.curl.dev/usingcurl/proxies/haproxy.html

Environment

VMware Tanzu Kubernetes Grid 1.x

Resolution

To Complete the above scenario (enable proxy protocol on both AVI and contour) the following steps have to be applied:

1. Configure AVI to use Proxy under L4 
Under AVI - Templates - Application - Create a new template with Enable proxy protocol v1 (do not select Preserve Client IP Address)


Associate this application profile with contour service.


2. Update installed contour to use Proxy 
Update the contour package with the settings below where useProxyProtocol: true:

cat datavalues.yaml

infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
  useProxyProtocol: true
  replicas: 2
  logLevel: info
envoy:
 service:
   type: LoadBalancer
   annotations: {}
   externalTrafficPolicy: Cluster
   disableWait: false
 hostPorts:
   enable: true
   http: 80
   https: 443
 hostNetwork: false
 terminationGracePeriodSeconds: 300
 logLevel: info
certificates:
 duration: 8760h
 renewBefore: 360h

Update package with below example:

tanzu package installed update contour --package contour.tanzu.vmware.com --version <VERSION> --values-file datavalues.yaml --namespace tkg-system

3. (Optional step) to add extra header using the httpproxy object:
Optionally you can enhance the httpproxy with additional header 
 

cat proxy.yaml

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: root
  namespace: projectcontour-roots
spec:
  routes:
  - conditions:
    - prefix: /
    requestHeadersPolicy:
      set:
      - name: X-Client-IP
        value: '%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%'
    services:
    - name: rootapp
      port: 80
  - conditions:
    - prefix: /secure
    services:
    - name: secureapp-default
      port: 80
  virtualhost:
    fqdn: my.firstcontour.io

Where the code added is:

    requestHeadersPolicy:
      set:
      - name: X-Client-IP
        value: '%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%'

As result test with curl command 
curl  http://FQDNOFHTTPPROXY:80 -v

contains the following headers where the IP I am running curl  from is 192.168.50.45:
              <li class="list-group-item">X-Client-Ip: [192.168.50.45]</li>
              <li class="list-group-item">X-Envoy-Expected-Rq-Timeout-Ms: [15000]</li>
              <li class="list-group-item">X-Envoy-Internal: [true]</li>
              <li class="list-group-item">X-Forwarded-For: [192.168.50.45]</li>
              <li class="list-group-item">X-Forwarded-Proto: [http]</li>