Command cf logs fails to retrieve logs from Log Cache: unexpected status code 502

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

After upgrading to cf CLI v6.50+, you might experience a 502 error when querying app's log with the cf log command or notice the Gorouter flushed with a huge increase of log-cache requests. 

Since cf CLI v6.50, the cf CLI streams logs from Log Cache instead of Traffic Controller - this is going to be deprecated. As a result of this change, cf logs fetches the current logs and forwards it to the terminal, it then sends another request immediately to fetch new logs. This behaviour is inside a loop without delay. As a result, Gorouter will be flushed with the requests caused by cf logs. It occurs even when the app is stopped. Thus a large number of requests to log-cache can be observed on Gorouter. Some of those requests may fail with a 502 error because log-cache failed to process those logs. 

REQUEST: [2021-04-30T10:24:45+08:00]
GET /api/v1/read/a28ecb62-####-####-8280-f8e87cd7dcd8?envelope_types=LOG&start_time=1619749481528602103 HTTP/1.1
Host: log-cache.run-07.slot-59.pez.vmware.com
--
REQUEST: [2021-04-30T10:24:46+08:00]
GET /api/v1/read/a28ecb62-####-####-8280-f8e87cd7dcd8?envelope_types=LOG&start_time=1619749481528602103 HTTP/1.1
Host: log-cache.run-07.slot-##.###.######.com
--

Before cf CLI v6.50, cf logs leverages WebSocket protocols, it starts one WebSocket session and receives log stream from the Traffic Controller.  

--
WEBSOCKET REQUEST: [2021-04-30T02:23:57Z]
GET /apps/a28ecb62-####-####-####-f8e87cd7dcd8/stream HTTP/1.1
Host: wss://doppler.run-07.slot-##.###.#####.com:443

Environment

Product Version: 2.10

Resolution

As a temporary workaround, please downgrade cf CLI to a lower version. For example, downgrade to cf CLI v6.46.