How to enable X-forwarded-for headers in an ingress controller in TKGi

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• You create a "network plan" and new cluster that allows forwarding of client IP addresses to app logs.
• This works great for an "ingress" service
• However, you're trying to get your "Ingress-Controller" to do the same thing:
  • It will not pass the Headers or IPs no matter what settings/annotations you use in the Ingress-controller config. This article works for setting up Ingress to individual apps, but it does not extend to Ingress controllers.  Your apps use nginx ingress and you need to get that to work with NSX + external IPs forwarding.

Environment

Tanzu Kubernetes Grid Integrated Edition, all supported versions

These docs provide background:

• Configure Your Kubernetes Ingress Resource to Use the New Ingress Load Balancer
• URL rewriting example 

Cause

The Nginx Ingress Controller (NIC) creates a service of type LoadBalancer for public access. This LoadBalancer service creates a VirtualServer in NSX, which is built under the NSX L4 LoadBalancer object associated with the cluster (as noted here). The decision to use the service of type LoadBalancer is coded into the NIC deployment or helm chart. This is something we can't change. The NSX L4 LoadBalancer object cannot utilize "x-forwarded-for" settings to forward client IP information to the downstream application. This is detailed in the following KB article.

Resolution

We know of 2 ways to accomplish this. 

1. Recommended: Disable the NSX-LB ingress controller entirely using a TKGi network profile, and rely on the third-party ingress controller for both the collection of X-forwarded-for headers and TLS termination. In the use case where X-forwarded-for is required along with TLS termination, the NSX-LB has more drawbacks than advantages.
   
   
2. One customer used a web-application-firewall (Fortinet) in conjunction with an NSX-T load balancer; the WAF appliance gives IP’s/Headers and encrypts the traffic between the WAF and app. The WAF terminates HTTPS while also encrypting the traffic.