IDM internal load balancer in Azure with disabled client IP in session persistence causes UI issues

Products

• CA Identity Manager

Issue/Introduction

Is there a recommended approach from Broadcom about Azure load balancer setup for Identity Manager?

We are in the process of troubleshooting an internal load balancer issue in our Production environment for Identity Manager v14.5.

The current setup is this:

End user -> hits the web server URL www.test.webserver.sample-> hosted on two Apache web servers -> which calls the internal load balancer, loadbalancer.test.sample-> load balancing 4 IDM clustered,  standalone application servers.

Currently, our setup in Production is hitting two main application servers as opposed to all four which should be evenly distributing the load. Based on a DB query of the runtimeStatusDetail12 table, we are seeing the following load distribution:

Row Labels    Count of serverName

IM_Server_01    429723
IM_Server_02    94533
IM_Server_03    68106
IM_Server_04    456213

Grand Total    1048575

Clearly, 01 and 04 are receiving the most traffic, and we are trying to identify where our issue is. We believe it is due to our internal load balancer loadbalancer.test.sample which is hosted on Azure Govt Cloud environment. Session persistence is set to Source IP on the Azure Portal, which uses a two-tuple hash (client IP only) setup, which causes all traffic from the same client IP to be consistently routed to the same backend instance. The Azure team reached out to Microsoft to understand if there was a better setup for load distribution and they mentioned to change load balancer rule distribution mode to none. 

The recommended approach from Microsoft was implemented in our lower environment, and unfortunately, when navigating through the IDM UI, we would sometimes be returned to the home screen, and we believe it is due to the session being handed off to another application server. Therefore, we do not think this is the best approach for our internal load balancer and would need some support to understand what we need to evenly distribute load across all four application servers.

They want to know if there is a recommended approach from Broadcom about Azure load balancer setup for Identity Manager?

We know if we can start by requesting some kind of network trace log to see how the traffic is behaving but we would like to know if you have any recommendations first.
NOTE: We already checked the documentation to ensure there were no Hotfix/Fixed Defects/Known Issues related to this specific problem.

Environment

IDM 14.x 

Cause

Lack of session stickiness set at the webserver/load-balancer tier.

The serverName row counts in runtimeStatusDetails12 table do not necessarily correlate with the front-end, incoming request activity to the individual cluster nodes because once the task is submitted, any node (and multiple nodes) in the cluster can process the task's events. 

Resolution

There is no special Identity Manager guidance for Azure webserver/load-balancing.

As with any web server or load balance solution that serves requests to Identity Manager, 

1. Enable session stickiness at the web server/load-balancer. 

2. Follow your application server's proxy/plug-in configuration requirements in conjunction with your webserver/load-balancer's requirements.

Consult your application server and/or webserver/loadbalancer support for additional assistance.