A2A calls on client connecting to VIP fail while cluster is turned off

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

We have a number of A2A clients configured to retrieve credentials with the setting "Use Server First" and a cache expiration of 3 days.  My expectation, and what seems to be in the documentation, is that the A2A agent will attempt to connect the PAM server and if it fails to connect the password from the cache will be used.

Last night I was performing some PAM maintenance that involved stopping the cluster.  During this time, A2A attempts for credentials failed instead of reverting to the cache.  From the timestamps it looks like this might have been during the cluster shutdown operation.  Is this expected behavior?

The A2A agents are configured to use the VIP of the primary cluster.

Information from one of the A2A logs is below:

WARNING: Wed September 09 21:01:25.677 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:01:25.677 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect
WARNING: Wed September 09 21:01:46.720 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:01:46.720 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect
WARNING: Wed September 09 21:02:07.763 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:02:07.763 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect
WARNING: Wed September 09 21:02:28.780 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:02:28.780 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect
WARNING: Wed September 09 21:02:49.814 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:02:49.814 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect
WARNING: Wed September 09 21:03:10.843 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:03:10.843 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect
WARNING: Wed September 09 21:03:31.862 EDT 2020 HttpService::httpsGetInputStreamFromConnectionArray. Https request failed, exception message: Connection timed out: connect
SEVERE: Wed September 09 21:03:31.862 EDT 2020 AccountService::processGetAccountRequest. HTTPS operation did not complete successfully. Connection timed out: connect

Environment

Release : 3.3.X and 3.4.X released before Jun 2021, and 4.0 GA.

Component : PRIVILEGED ACCESS MANAGEMENT

Cause

This was caused by an implementation flaw in the A2A client. The timeout for the local API calls into the A2A client service was shorter than the timeout for the connection to the PAM server, and therefore the A2A call failed before the A2A service would have switched from server to local cache for credential retrieval. The same problem would be observed if the A2A client connected to a specific PAM server and that server was down.

Resolution

This problem is fixed in the latest A2A client versions, starting with 3.4.4, which was released after PAM 4.0. See the following item at the bottom of page Resolved Issues in 3.4.4:

 

32218912 DE490143 A2A client configured with cluster VIP fails to provide cached passwords while cluster is off

Additional Information

Note that all recent A2A client versions are based on the 4.12.3 version and will be listed with that same version in PAM. The 3.4.4 version uses Java version 1.8.0_292. Any A2A client using this Java version, or a newer version, will have the fix included. E.g. to check the version on a Linux server, where the client is installed into /opt, you would see the following version information for a 3.4.4 A2A client:

> /opt/catech/cspmclient_thirdparty/java/bin/java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)