Error PAM-CMN-5199 The cluster configuration has changed on XXXX. adding node to CA PAM Cluster

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

After removing a node from a cluster site to perform some maintenance, dynamically adding it back is not possible as the following message is displayed

Error: PAM-CMN-5198: Failed to join the cluster. PAM-CMN-5199: The cluster configuration has been changed on XXXX. Please re-download and try again.

Where cluster member XXXX is not even from the same cluster site the node is being added to, but from a completely different site

Cause

This error is a bit misleading, because it does not correspond to the real cause for the problem.

When the process of adding a node to a cluster is initiated, the node queries the rest of the nodes in the cluster (in its own site and elsewhere) to obtain their configuration. 

If for some reason communications are blocked to one of the other nodes, this message will be displayed.

In the php_error.log a message similar to the following will be present:

[ 10:31:28 09/20/21 ] [ error ] [Request-614862f558238]:  CURL request to scheme=https&host=XXX.XXX.XXX.XXX&port=8443&path=%2Fajax_cmd.php&query=cmd%3DACTACT%26cmdtype%3DGETCONFS returned error (7):  Failed to connect to XXX,XXX.XXX.XXX port 8443: Connection refused [ /var/www/htdocs/uag/hconfig/functions/failover_functions.php : 59 ]

Environment

CA PAM 3.3.X, 3.4.X, 4.X

Resolution

Make sure port 8443 is open between the different cluster sites both ways, as specified in the documentation:

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/4-0-1/deploying/set-up-a-cluster/cluster-deployment-requirements.html

See section TCP/Clustered appliances in the above document