It is possible to configure CA PAM to stay behind an F5 or similar device acting as an HTTP Proxy. When this happens, connection to CA PAM is possible, but any attempt at creating an SSH or RDP connection from PAM, be it by using the Applet or by defining a service, invariably fails
However, if the F5 or equivalent machine is configured not to work as an HTTP proxy, and it is configured with the right certificate at F5 and ssl decryption (for compliance purposes), all is working fine
Release : 4.X
The reason for this is that when configured as an HTTP reverse proxy, F5 would expect HTTPS-encapsulated traffic, meaning it would expect an https-like structure, complete with headers.
When SSH or RDP access is launched, PAM launches an applet, and that is the one that does the connection. This is an openssl connection directly (if wireshark is installed negotiation and handshake are visible) in the case of SSH and using RDP (with or without NLA) in the case of Windows. What happens is that PAM opens a local port (e.g. 127.0.0.200:12345) through which the traffic tunnels to PAM and to the remote target server. The SSH or RDP applet will connect to that port.
Thus what goes through the network and to the target system (Windows or Linux, or even Webportal) is TCP traffic, but not necessarily https-format traffic. It will correspond to and use the protocols available for encrypted SSH or RDP traffic.
If the https proxy is inspecting traffic and just allowing traffic in https format, that explains the situation. And also why establishing a service without RDP set up as protocol is also failing.
When a TCP service is established, for instance without choosing RDP as protocol, what happens is that PAM does not try to do anything with the traffic (as opposed to what happens when you choose protocol RDP or SSH, for instance): it just creates a tunnel and takes the stream of data through it and onto the remote system.
So, if the http proxy is expecting https-formatted traffic it may be once again experiencing the same issue.
There is no resolution on PAM side, since as of now it is not sending data encapsulated in https. There may be some solution on F5 side so that non-https format is allowed in through the proxy, but this should be discussed with their support