In PROD environment TEWS calls are taking time to respond back. In the user console there is also extreme slowness logging in and navigating tasks elsewhere in the environment.
The problem occurs in only one of two Identity Manager environments (IMEs). The main difference between the two IMEs is that the Active Directory user stores are in completely separate, unrelated servers.
We isolated the issue to User Store AD LDAP performance--in the logs there were consistently long 20-30 plus seconds gaps after acquiring LDAP connections until a return.
Also, there were no performance issues with the other IME which points to AD user store on a different server.
On the AD side, once a query reached AD it was processed quickly, but the AD team determined that referrals should be disabled but were enabled in SSO.
The following tuning particularly disabling AD referrals for the AD user resolved the performance issue.
Update a Policy Server registry entry:
EnableReferrals = 0 (disabled).
Then on the IDM side, updated the directory.xml to ignore referrals plus the following tunings:
<PropertyDict name="LDAP_CONNECTION_SETTINGS">
<Property name="com.sun.jndi.ldap.connect.pool.maxsize">200</Property>
<Property name="com.sun.jndi.ldap.connect.pool.prefsize">10</Property>
<Property name="com.sun.jndi.ldap.connect.timeout">5000</Property>
<Property name="java.naming.referral">ignore</Property>
</PropertyDict>
After restarting the IME, performance in the IME became normal again, with nearly instant response time in a variety of task activities.