PAM internal VIP logic to distribute load in a cluster
Article ID: 368484
Updated On:
Products
Issue/Introduction
It is important to understand how the PAM cluster VIP manages connections when deciding whether to utilize the internal VIP or an external 3rd party load balancer.
Environment
CA Privileged Access Manager (PAM) cluster implementations with floating IP VIPs.
Resolution
The PAM Cluster VIP is assigned to the primary cluster node at the time the cluster is started. Specifically, this IP is assigned as a second IP to the network interface defined in the cluster configuration. This requires that the VIP is defined with an IP that can be routed through the network interface it is assigned to. When planning the cluster site configuration, it is important to ensure that all nodes within a specific cluster site can make use of that network IP on the network device defined.
Depending on the type of network traffic coming into the PAM VIP, the built-in load balancer may route traffic to any node within the cluster site. Using the built-in VIP, you cannot have cluster traffic balanced to any other site. If you need to load balance client connections to other sites within a multi-site cluster you need to use a 3rd party load balancer configured separately from the internal VIP address. All client connections coming into the VIP address will be routed to the local address of one of the cluster nodes within the site after determining the least utilized node. Utilization is determined only by the number of active “xcd_spfd” processes on each node. An xcd_spfd child process is started for every client connection plus each active device connection. Once a client connection is made to a PAM appliance, all communication from that client will remain on that PAM appliance until it is logged off. This is true regardless of using the built-in VIP or a third-party load balancer VIP.
Example:
The first connection made to a 3-node cluster site will go to the cluster site node owning the VIP, since there are an equal number of connections. Assume that two more connections come in and are distributed to the other two nodes. Then each of the PAM users connected to the other two nodes starts three connections (access sessions) to target devices. When the next 3 client connections come in, all 3 connections will go to the first node. PAM does not display the number of xcd_spfd processes anywhere in the UI, but the number of user sessions on UI page Sessions > Manage Sessions plus the sum of connections listed on this page under column Connections should be close to the load number that the internal VIP calculates. There could be additional load from API or A2A calls as well as from internal cluster communication.
Additional Information
External load balancers can be configured to balance the load by the number of external client connections, but cannot use the load balancing methodology described above, since they cannot determine the number of access sessions launched by each user. Most third party load balancers will use a round robin with source persistence methodology to manage the load in a simple manner.
Aside from client connections, no other types of connections into port 443 are load balanced using the built-in VIP. Internal cluster communication, A2A password requests, and API or CLI commands will spawn a new xcd_spfd child process for the duration of the call, but these will not be redirected to any other node in the cluster site. It is possible to have a higher load on the primary cluster node when all or most of these additional communications go to the VIP.
Feedback
