Spring Boot app returns HTTP status code 302 in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When accessing a Spring Boot app on Tanzu Application Service (TAS) for VMs through a browser or REST API, the app returns the HTTP status code 302 and redirects to itself.

The embedded Tomcat in a Spring Boot app is configured with a default regular expression that matches internal proxies to be trusted so that it can process the x-forwarded-for header. By default, IP addresses in 10/8, 192.168/16, 169.254/16, 127/8, 172.16/12, and ::1 are trusted. 

In TAS for VMs v2.2 or lower and TAS for VMs v2.3, when Router application identity verification is not enabled, Gorouters forward traffic to apps directly as external proxies.

When TAS for VMs is installed in a non-RFC1918 network, this regular expression does not match. As a result, Tomcat does not trust any of the Gorouter IPs.

This might not seem important, but when this happens, Tomcat also does not process the x-forwarded-proto header and the application will not know if the incoming request is over HTTP or HTTPS. It defaults to HTTP when http.requiresChannel().anyRequest().requiresSecure() is set in the Spring Boot app. The app redirects you to HTTPS and Tomcat returns the HTTP status code 302. This happens indefinitely because the request is actually coming in through HTTPS and the app does not know.

Proper configuration of the regular expression for the x-forwarded-for header will result in Tomcat processing the x-forwarded-proto as well.

Resolution

To work around this issue, determine a proper IP address range from your TAS for VMs foundation and make all Spring Boot apps recognize this range as trusted proxies with 'cf set-running-environment-variable-group'.

In the case Router application identity verification is not enabled, the IP addresses range should include TAS for VMs Gorouter IPs.

In the case Router application identity verification is enabled, the IP addresses range should include TAS for VMs Overlay Subnet. This is configured at PAS > Networking > Overlay Subnet and is mandatory for TAS for VMs 2.4+ when TLS to verify application identity enabled, by default it's 10.255.0.0/16. 

For test purposes against a single app, cf set-env is an option. For example, if Gorouter IPs are at 100.xxx.xxx.xxx range and Router application identity verification is enabled, apply the trusted proxies for Spring Boot apps across the foundation with the following command:

$ cf srevg '{"SERVER_TOMCAT_INTERNAL_PROXIES":"100\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|192\\.168\\.\\d{1,3}\\.\\d{1,3}|169\\.254\\.\\d{1,3}\\.\\d{1,3}|127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|172\\.1[6-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3}|0:0:0:0:0:0:0:1|::1", "SERVER_TOMCAT_REMOTEIP_INTERNAL_PROXIES":"100\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|192\\.168\\.\\d{1,3}\\.\\d{1,3}|169\\.254\\.\\d{1,3}\\.\\d{1,3}|127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|172\\.1[6-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3}|0:0:0:0:0:0:0:1|::1"}'

The reason to set both SERVER_TOMCAT_INTERNAL_PROXIES and SERVER_TOMCAT_REMOTEIP_INTERNAL_PROXIES is until Spring Boot 2.2.9 release, the property is server.tomcat.internal-proxies, while since Spring Boot 2.3.0 release, the property has been changed to server.tomcat.remoteip.internal-proxies. 

To test against a single app with the following command:

$ cf set-env APP_NAME SERVER_TOMCAT_INTERNAL_PROXIES '100\.\d{1,3}\.\d{1,3}\.\d{1,3}|10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}| 172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}|0:0:0:0:0:0:0:1|::1'

$ cf set-env APP_NAME SERVER_TOMCAT_REMOTEIP_INTERNAL_PROXIES '100\.\d{1,3}\.\d{1,3}\.\d{1,3}|10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}| 172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}|0:0:0:0:0:0:0:1|::1'

Note: The app must be restarted before the configuration takes effect. 

In TAS for VMs v2.4 and above releases and TAS for VMs v2.3, when Router application identity verification is enabled between Gorouter and apps, an envoy process in the app container acts as app external proxy, because container overlay subnet is required to be in RFC1918 network, above SERVER_TOMCAT_INTERNAL_PROXIES/SERVER_TOMCAT_REMOTEIP_INTERNAL_PROXIES can be removed.

If SERVER_TOMCAT_INTERNAL_PROXIES/SERVER_TOMCAT_REMOTEIP_INTERNAL_PROXIES is configured as Gorouter IPs range only (without RFC1918 range) before, SERVER_TOMCAT_INTERNAL_PROXIES/SERVER_TOMCAT_REMOTEIP_INTERNAL_PROXIES must be removed or configured to include container overlay subnet. 

Additional Information

When applying this workaround, the value of SERVER_TOMCAT_INTERNAL_PROXIES completely overrides the tomcat defaults external proxies of 10/8, 192.168/16, 169.254/16, 127/8, 172.16/12, and ::1. If you require trusting some private and some public networks, you must list them all in SERVER_TOMCAT_INTERNAL_PROXIES.