ERR_TOO_MANY_REDIRECTS is returned when accessing Apps Manager

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

TAS is on version 2.11.29 and suddenly ERR_TOO_MANY_REDIRECTS error is returned when accessing Apps Manager, which means the request has been redirected for too many times.
301 status code is also found in Apps Manager app logs.

 2023-06-14T02:42:47.27+0800 [APP/PROC/WEB/0] OUT #.#.#.#, y.y.y.y - https://apps.exmaple.com/ - [13/Jun/2023:18:42:47 +0000] "GET / HTTP/1.1" 301 162

This issue could be temporarily relieved by setting environment variable SKIP_PROTOCOL_VALIDATION to true for Apps Manager app. However "App Metrics" link is still missing on Apps Manager even with that temporary solution.

And 301 status code is still returned when accessing endpoints of search-server application.

There is F5 load balancer (LB) sitting in front of router instances on this TAS foundation and it's working at layer 7. When accessing search-server app through LB, 301 status would be returned.

$ curl -v -k https://search-server.<sys domain>/proxy/home/app-metrics-available
* About to connect() to search-server.<sys domain> port 443 (#0)
......
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 118
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 15 Jun 2023 05:41:33 GMT
< Location: https://search-server.<sys domain>/proxy/home/app-metrics-available
< Vary: Accept
< X-Powered-By: Express
< X-Vcap-Request-Id: 33c9d033-aa93-4b27-76fb-6a1410f77b80

But same request would get 500 (which is expected) if accessing through gorouter directly which means the LB is bypassed.

# curl -v -k -H "Host: search-server.<sys domain>" https://<router_IP>/proxy/home/app-metrics-available
......
< HTTP/1.1 500 Internal Server Error
< Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate
< Content-Length: 148
< Content-Security-Policy: default-src 'none'
< Content-Type: text/html; charset=utf-8
< Date: Thu, 15 Jun 2023 05:39:28 GMT
< Expect-Ct: max-age=0
< Expires: 0
< Pragma: no-cache
< Referrer-Policy: no-referrer
< Strict-Transport-Security: max-age=15552000; includeSubDomains
< Surrogate-Control: no-store
< X-Content-Type-Options: nosniff
< X-Dns-Prefetch-Control: off
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Vcap-Request-Id: 1517df8a-6ab8-4f3d-52fd-bfbfbd577030
< X-Xss-Protection: 0
<
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Internal Server Error</pre>
</body>
</html>

Since the system has been running well then such issue suddenly happens, it means some change has been made at F5 LB side which has caused this symptom.

Environment

Product Version: 2.11

Cause

If you check on the F5 LB end, you will find that the following rule, which is mentioned in Configuring an F5 load balancer for TAS for VMs, has been removed from the virtual server by mistake.

Name: cf-xforward-proto-http
Definition: when HTTP_REQUEST { HTTP::header insert X-Forwarded-Proto "http" }

Resolution

Add this rule back to the virtual server in F5.