When a LAYER7 server was recycled, the connection to CALDAP on the mainframe was lost and could not be re-established. The connection was only re-established after CALDAP was cancelled and restarted. The only error seen is Layer7 reporting that it is unable to reach CALDAP:
2020-02-25T15:50:34.482-0500WARNING 521 com.l7tech.server.identity.ldap.BindOnlyLdapUserManagerImpl:Couldnot establish contexton any of the ldap urls.
Release : 16.0
Component : CA LDAP Server
All of the CALDAP threads were waiting for a connection response. The LAYER7 server is what is used to connect to LDAP. When the LAYER7 server was recycled, those connections were still waiting for responses. So after the LAYER7 server came back up, there were no more threads available to make a new connection to CALDAP.
All the threads were in the IBM tls gsk handshake code waiting for a response that never came.
Restarting the LDAP resolver after the LAYER7 server was back up freed up the threads so a new connection could be made.
The IBM doc does not specify any handshake timeout value for z/OS. For ISeries, the default is to wait forever. ISeries allows the timeout value to be changed, but z/OS does not.
IBM has something called AT-TLS that allows any application to use TLS over normal TCP connections. This has a handshake timeout. AT-TLS could be configured on the mainframe where LDAP is, then use the normal 389 port from the LAYER7 server to connect to LDAP. If this is something that continues to be a problem, switching to AT-TLS is an option.
Increasing the threads (in the LDAP slapd.conf file) will allow for more connections, but at best this is just pushing the problem down the road. At worst, not enough threads would be available to do work and things could run slower.