Symptom: After upgrading to PAM 2.8.1; customer is not able to launch any RDP/SSH connections target servers.
Error: Existing connection was forcefully closed by the remote host due to time out while connecting to any backend servers through PAM
Engineering Notes: Being the cluster is up and ALL GREEN for Access and PA, then the UPD process may have stopped and needs to be restarted. Assuming that that node that is unresponsive with RDP error not being able to reach the backend device, all we needed to do is login through SSH and restart the process at the command prompt: xcd_upd. The usual workaround for this problem of disabling the Retrieve IP address feature from Global Setting and restarting UPD didn’t work.
Root cause: The problem is due to some of the users accessing PAM running an older version of the PAM Access Agent (UP applet) that is incompatible with some of the Threat Analytics enhancements that are part of 2.8/2.8.1. The code is expecting all users of PAM to be running 2.8.1 version of the PAM applets but we suspect due to jar caching that is not the case, there are some users that are connecting with 2.7 based jars. Code change was made to the primary node; it was to deploy a debug daemon that helped us pinpoint the problem. There was no need to deploy it on the secondary also because we could readily reproduce the problem on the primary.
Permanent Fix: Hot Fix Patch PAM 2.8.1.03.
Note: Patch is only required for 2.8.1. If a customer is on 2.8, they’ll have to upgrade first.