AVI Load Balancer and Package Contour/Envoy do not preserve Client IP in X-Forwarded-For

Products

• VMware Tanzu Kubernetes Grid
• VMware vSphere Kubernetes Service

Issue/Introduction

When TKGm/TKGs is configured with AVI and contour package the following configuration takes place:

• When contour is created it uses a loadbalancer service which is realized as virtual service in AVI, once service and deployment is completed we have to configure httpproxy (ingress) so any of the pages registered to be forwarded to the respective K8s Application (cluster services)
• Some applications require information about the client who is reaching the endpoint, and for this reason, they require to record the client IP address using http headers like "X-Forwarded-For"
• Because AVI Virtual service is configured as an L4 loadbalancer there are two possible scenarios
• In L4 mode, preserving client IP requires Avi to be configured with either proxy protocol or native client IP preservation, both of which require an Avi Enterprise license.

Since Avi is doing L4 only in this case, it can't set the X-Forwarded-For header as this option is only for HTTP L7 Virtual service, so that is not an option here.

Contour can be configured to operate using any of the above approaches:

• If using PROXY protocol, this needs to be enabled in Contour via the `contour.useProxyProtocol` package setting 
• If using X-Forwarded-For (would require Avi to be doing L7), Contour needs to be configured with the correct number of trusted hops via `contour.configFileContents.network.num-trusted-hops` package setting as discussed here:
  • Contour for Ingress Control Optional Configurations
  • Network Configuration
  • This option is not available with the service of type L4.

 

If Contour is configured to use PROXY protocol then any packets that are not coming with PROXY protocol will be dropped so if a Pod within the cluster wants to access such service it will have to use PROXY protocol, which can be done via the flag curl --haproxy-protocol https://example.com/. 

You can find more information on the following: https://everything.curl.dev/usingcurl/proxies/haproxy.html

Environment

• VMware Tanzu Kubernetes Grid (TKGm)
• vSphere with Tanzu (TKGs)

Resolution

To Complete the above scenario (enable proxy protocol on both AVI and contour) the following steps have to be applied:

1. Configure AVI to use Proxy under L4 
Under AVI - Templates - Application - Create a new template with Enable proxy protocol v1 (do not select Preserve Client IP Address)


Associate this application profile with contour service.


2. Update installed contour to use Proxy 
Update the contour package with the settings below where useProxyProtocol: true:

cat datavalues.yaml

infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
  useProxyProtocol: true
  replicas: 2
  logLevel: info
envoy:
 service:
   type: LoadBalancer
   annotations: {}
   externalTrafficPolicy: Cluster
   disableWait: false
 hostPorts:
   enable: true
   http: 80
   https: 443
 hostNetwork: false
 terminationGracePeriodSeconds: 300
 logLevel: info
certificates:
 duration: 8760h
 renewBefore: 360h

Update package with below example:

tanzu package installed update contour --package contour.tanzu.vmware.com --version <VERSION> --values-file datavalues.yaml --namespace tkg-system

3. (Optional step) to add extra header using the httpproxy object:
Optionally you can enhance the httpproxy with additional header 
 

cat proxy.yaml

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: root
  namespace: projectcontour-roots
spec:
  routes:
  - conditions:
    - prefix: /
    requestHeadersPolicy:
      set:
      - name: X-Client-IP
        value: '%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%'
    services:
    - name: rootapp
      port: 80
  - conditions:
    - prefix: /secure
    services:
    - name: secureapp-default
      port: 80
  virtualhost:
    fqdn: my.firstcontour.io

Where the code added is:

    requestHeadersPolicy:
      set:
      - name: X-Client-IP
        value: '%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%'

As result test with curl command 
curl  http://FQDN_OF_HTTP_PROXY:80 -v

contains the following headers where the IP I am running curl  from is 192.168.50.45:
              <li class="list-group-item">X-Client-Ip: [192.168.50.45]</li>
              <li class="list-group-item">X-Envoy-Expected-Rq-Timeout-Ms: [15000]</li>
              <li class="list-group-item">X-Envoy-Internal: [true]</li>
              <li class="list-group-item">X-Forwarded-For: [192.168.50.45]</li>
              <li class="list-group-item">X-Forwarded-Proto: [http]</li>