Attributes Fail to Sync - code 52 / error code 70

Products

CA Identity Manager CA Identity Suite

Issue/Introduction

Endpoint attribute changes made via CA Identity Manager fail to synchronize and report errors like the ones below in the provisioning server (etatrans.log) logs and the application server (server.log) logs.

Connector Server Modify failed: code 52 (UNAVAILABLE)

LDAP: error code 70 - :ETA_E_0039<MGU>

For example:

etatrans

20200119:222412:TID=002524:Modify :C066:E039:F: FAILURE: Child Modify (eTDYNAccountName=TESTU)
20200119:222412:TID=002d40:EtaServer :----:----:I: Retrieving common BLS Connectivity Configuration
20200119:222412:TID=002524:Modify :C066:E039:F: rc: 0x0034 (DSA is unavailable)
20200119:222412:TID=002524:Modify :C066:E039:F: msg: :ETA_E_0008<MAC>, User Account 'TESTU' on 'S3APP' modification failed:
20200119:222412:TID=002524:Modify :C066:E039:F:+ Connector Server Modify failed: code 52 (UNAVAILABLE): failed to modify entry: e
20200119:222412:TID=002524:Modify :C066:E039:F:+TDYNAccountName=TESTU,eTDYNAccountContainerName=Accounts,eTDYNDirectoryName=123
20200119:222412:TID=002524:Modify :C066:E039:F:+45,eTNamespaceName=12345,dc=im,dc=etasa: JCS@DC01: doLookUpobjectClass=
20200119:222412:TID=002524:Modify :C066:E039:F:+'eTDYNAccount', name='TESTUSER', containerName='eTDYNAccountContainerName=Accounts
20200119:222412:TID=002524:Modify :C066:E039:F:+': com.ca.jcs.jdbc.JDBCMetaConnector: 12345: exhausted 6 retries without reconnec
20200119:222412:TID=002524:Modify :C066:E039:F:+ting: org.apache.directory.shared.ldap.exception.LdapServiceUnavailableException:
20200119:222412:TID=002524:Modify :C066:E039:F:+ JCS@DC01: JDBC: org.springframework.jdbc.CannotGetJdbcConnectionExcept
20200119:222412:TID=002524:Modify :C066:E039:F:+ion: Could not get JDBC Connection; nested exception is java.sql.SQLException: JZ
20200119:222412:TID=002524:Modify :C066:E039:F:+00L: Login failed. Examine the SQLWarnings chained to this exception for the rea
20200119:222412:TID=002524:Modify :C066:E039:F:+son(s). (ldaps://dc01.server.com:20411)

Server Log Entries

22:23:49,914 ERROR [ims.llsdk.directory.jndi] (Thread-3546 (HornetQ-client-global-threads-1425241877)) Failed to modify global user/accounts:[LDAP: error code 70 - :ETA_E_0248<MGU>, Global User 'TESTU' status updated successfully. Associated account status update failed: (accounts updated: 2, unchanged: 0, failures: 1) ]

22:23:49,914 ERROR [ims.llsdk.directory.jndi] (Thread-3546 (HornetQ-client-global-threads-1425241877)) Failed to modify managed object ObjectType::USER with unique name eTGlobalUserName=TESTU,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta Error message: [LDAP: error code 70 - :ETA_E_0039<MGU>, Global User 'TESTUSER' updated successfully. Associated accounts update failed: (accounts updated: 0, unchanged: 2, failures: 1) ]

22:23:49,914 ERROR [im.provisioning] (Thread-3546 (HornetQ-client-global-threads-1425241877)) [LDAP: error code 70 - :ETA_E_0039<MGU>, Global User 'TESTUSER' updated successfully. Associated accounts update failed: (accounts updated: 0, unchanged: 2, failures: 1) ]

22:23:49,929 ERROR [com.netegrity.ims.exception.EventExecuteStateException] (Thread-3546 (HornetQ-client-global-threads-1425241877)) Execution of event: SynchronizeAttributesWithAccountsEvent failed. Exception encountered: [LDAP: error code 70 - :ETA_E_0039<MGU>, Global User 'TESTUSER' updated successfully. Associated accounts update failed: (accounts updated: 0, unchanged: 2, failures: 1) ]

22:23:49,929 ERROR [com.netegrity.ims.businessprocess.IMSEventController] (Thread-3546 (HornetQ-client-global-threads-1425241877)) Error during event execution [beb7ad84-0a21c194-5302bf88-f1e9b368] SynchronizeAttributesWithAccountsEvent

Environment

Release : 14.x

Component : IdentityMinder(Identity Manager)

Cause

This issue occurs as a result of connectivity issues between the Provisioning Server and the Connector Server servicing the endpoint.

Resolution

Make sure all endpoints are available and accessible and restart the Connector Server if necessary.

Additional Information

To configure the KeepAlive parameters, use Regedt32.exe to modify the system registry. Be forewarned that this is a dangerous thing to do (if done incorrectly). You might want to consult with Microsoft before making any changes, as improper registry changes can cripple your system.

->> Add or modify the following registry keys under (Reboot once completed so new settings can take effect):

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters:
KeepAliveInterval: REG_DWORD = Interval between KeepAlive packets when one is dropped (Measured in milliseconds--the default is 1000 (1 second)).
KeepAliveTime: REG_DWORD = How often TCP attempts to verify the connection Measured in milliseconds--the default is 7200000 (2 hours)
TcpMaxDataRetransmissions: REG_DWORD = Number of retries TCP does for a packet before giving up. The default is 5.

For MQ, reasonable values for these parameters might be 1000, 300000, and 5, respectively. This would have TCP/IP wait five minutes for data, then send up to five KeepAlive packets, one per second, to verify the connection. If the remote side does not respond, TCP/IP will end the link.
Enable KeepAlive for MQ v6 and v7 on Windows

We have seen this type of problem in the past and making changes to the Tcpip parameters helped, and we have some tuning that can be used to adjust the connections into and out of Provisioning Server.
We would recommend you test this in a DEV or QA or UAT environment first to ensure no other issues appear.

->> Other additional changes that can be added to the Provisioning / Connector server:

Listing 1. Registry changes
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpTimedWaitDelay"=dword:0000001e
"MaxUserPort"=dword:0000fffe
"TcpWindowSize"=dword:0000ffff
"MaxFreeTcbs"=dword:00011940
"MaxHashTableSize"=dword:0000ffff

These tuning parameters cover several aspects of the Windows operating system. Let's discuss them one by one.

The TcpTimedWaitDelay parameter, which controls the amount of time the operating system waits to reclaim a port after an application closes a TCP connection, has a default value of four minutes. With heavy loads, these limits might be exceeded and so can result in an address in use: connect exception. If you experience address in use: connect exceptions, try to increase the value of MaxUserPort and TcpTimedWaitDelay in the Registry.

The MaxUserPort parameter determines the highest port number that TCP can assign when an application requests an available user port from the system. Typically, ephemeral ports (that is, those used only briefly) are allocated to port numbers 1024 through 5000.
To increase the throughput of the server, decrease TcpTimedWaitDelay from the default of four minutes to 30 seconds (0000001e), and increase MaxUserPort from the default of 5000 to 65,534 (0000fffe) as shown in listing 1.

The TCPWindowSize parameter sets the maximum TCP window. With the large TCP window size, fewer acknowledgments are sent back, resulting in more optimized network communication between the sender and the receiver. Change the default of 17,520 bytes to 65,535 (0000ffff ) bytes as shown in listing 1.

The MaxFreeTcbs parameter is the threshold number of active TCP connections required before TCP Control Blocks in the TIME-WAIT state are reused. Changing MaxFreeTcbs allows the system to avoid reusing TCP Control Blocks. We changed the value of MaxFreeTcbs from the default of 1m000 to 72,000 (00011940) as shown in listing 1.

The MaxHashTableSize parameter controls the size of the TCP Control Block (TCB) table. The TCB table stores the control values for each active TCP connection. On a server with many active network connections, a large TCB table can reduce the amount of time that the system spends to locate a particular TCB. We changed the value of MaxHashTableSize from the default of 512 (0X200) to 65,535 (0000ffff) as shown in listing 1.

Apart the Windows TCPIP parameters afore mentioned, you can also check the following.

->> JIAM session pool for multiple connections to the Provisioning Server :

Enable Session Pooling
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-5/reference/advanced-configuration-options/domain-configuration/processes-parameters.html

To improve performance, CA Identity Manager can preallocate a number of sessions for pooling when communicating with the Provisioning Server.
If the Session Pools option is disabled, CA Identity Manager creates and destroy sessions as needed.
For a new environment, Session Pools are enabled by default. For existing environments, you can enable Session Pools.
Follow these steps:
1. In the Management Console, choose Advanced Settings, Provisioning.
2. Select Use Session Pool.
3. Define the minimum number of sessions in the pool at startup.
4. Define the maximum number of sessions in the pool.
5. Click Save.
6. Restart the Application Server.
The Session Pool is enabled per the defined settings.
Note: CA recommends setting the initial sessions value to 8, and the maximum sessions to 128.

->> Provisioning Domain Configuration settings :

Provisioning Domain Configuration settings, Connections Parameters:
https://docops.ca.com/ca-identity-manager/14-3/EN/reference/advanced-configuration-options/domain-configuration/connections-parameters

- Other Pool Minimum Size, default '0'
- Other Pool Maximum Size, default '20'
These can be increased, I would suggest small steps, increasing the Max to 50 and testing to ensure other performance issues do not appear.

- Expiration Time, default '1800' seconds (30 minutes)
This could be candidate to tune and decrease to allow the conections to be resued more quickly. However I would not recommend less than '300' seconds (5 minutes).

- Refresh Time, default '300' seconds (5 minutes) value ?
Maybe this could be candidate to tune and decrease. I would not recommend less than '180' seconds (3 minutes).

-> Processes Parameters
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-5/reference/advanced-configuration-options/domain-configuration/processes-parameters.html

- Child Operation Thread Pool Size, default '200'
- Parallel Propagation, default 'Yes'

Parallel propagation should be set to YES. Child operations can be increased, again in relatively small increments, from 200 to 500, and test to verify no other issues appear.

If you still continue to experience issues even after enabling the keep-alive and suggestions, this would likely indicate the issue is on the Networking / Active Directory side.