Troubleshooting the "EXCEPTION(invalid_request): Request could not be handled
Last Error: URL could not be parsed for matching purposes '' failed because: Scheme was not delimited by '://'"
Having, independently, investigated the uploaded policy trace debug and PCAP, we see the below.
What we see in the policy trace debug is an exact match with the capture seen in the PCAP and it goes on, in cycles (loop), as the 172.21.75.71 sends the web request, for the "Unknown URL" to the Proxy (172.21.46.25). See snippet below.
We see, from the policy trace, that the web request from the 172.21.75.71 reached the ProxySG appliance (172.21.46.25) with an unknown URL, as seen in the trace. This returned the "(invalid_request): Request could not be handled" exception, with the eventual "400" client response code. The root cause is also seen in the trace, which is " Last Error: URL could not be parsed for matching purposes '' failed because: Scheme was not delimited by '://'". Looking at the checkpoint timings for the trabnsaction, we do not see the "client-in: start x elapsed y ms". See the below.
Ref. doc.: https://knowledge.broadcom.com/external/article/168320/understanding-policy-trace-checkpoint-ti.html
The Proxy never get's to receive a URL request from the client, even though it received TCP connections from the client that should have culminated in a CONNECT request. Because of the invalid URL, lacking the "://", ProxySG returns the "400" response code to the client.
That's exactly the same behavior seen in the PCAP stream. After the ProxySG appliance, in frame 5, sends the last ACK, to complete the TCP handshake, it expected a HTTP CONNECT request in frame 6. That did not happen because of the invalid URL. With that, in frame 7, the ProxySG appliance sends a FIN ACK, to request the termination of the packet. In frame 8, the client machine resets the connection.
Significantly, because of the invalid URL request, URL categorization/policy evaluation did not happen and that's why we can see a litany of "late" tags in the policy trace.
Note: The late tag in a policy trace means that the ProxySG appliance reached a verdict on the connection before it could evaluate the policy rules that otherwise would have been evaluated later in the connection. See Tech. Doc. with URL below. The late tag appeared multiple times in a policy trace because the ProxySG appliance terminated the transaction early in (before) the evaluation. In this case, all of the policy rules that were not evaluated are considered late. This went on, in cycles (loops), in the policy trace, just as it's seen in the PCAP.
https://knowledge.broadcom.com/external/article/169002/in-a-policy-trace-what-does-the-late-tag.html
From the logs, we see, clearly, that the web requests failed not because of the web request but because of the invalid request, with invalid URL, sent by the client.