Intermittent slow response, Gen client/629 error and TIML SOCKET CLOSED

Products

• Gen
• Gen - Run Time Distributed

Issue/Introduction

Gen 8.6 application users were reporting intermittent occurrences of slow response mostly ending eventually with a 629 error message 

The application is using direct TCP/IP to CICS on z/OS connecting via Client Manager to the Multi Socket Listener TISRVMSL (trancode TIML).

These messages mentioning the TIML listener trancode appeared in CICS (CICR) logs at high frequency every few minutes during the time the slow responses were happening.
Want to check what the implication of these are with regards to the behavior reported and whether it gives any information/clue what could be causing the slow response:

TISRVMSL TIML TASK=00000048 02/07/2022 22:10:21 CONNECTION WITH NO DATA, UNSUCCESSFUL RETRY, SOCKET CLOSED  
TISRVMSL TIML TASK=00000048 02/08/2022 01:23:47 CONNECTION WITH NO DATA, UNSUCCESSFUL RETRY, SOCKET CLOSED   

TISRVMSL TIML TASK=00000048 11/29/2021 22:12:43 CONNECTION WITH NO DATA, UNSUCCESSFUL RETRY, SOCKET CLOSED 
TISRVMSL TIML TASK=00000048 11/29/2021 22:12:43 CONNECTION WITH NO DATA, UNSUCCESSFUL RETRY, SOCKET CLOSED 
TISRVMSL TIML TASK=00000048 11/29/2021 22:14:17 CONNECTION WITH NO DATA, UNSUCCESSFUL RETRY, SOCKET CLOSED 
TISRVMSL TIML TASK=00000048 11/29/2021 22:15:40 CONNECTION WITH NO DATA, UNSUCCESSFUL RETRY, SOCKET CLOSED 

Resolution

Error 629 means TIRM629E which represents a general client/server communication problem.

The problem only occurs intermittently and only at low usage times of day outside of normal business hours, per the examples of times in the CICS log above.
The TIML timeout values in EZACONFG should be adequate to prevent the normal problems reported for these symptoms i.e.
GIVTIME      ===> 060          Timeout Value for GIVESOCKET      
REATIME      ===> 060          Timeout Value for READ     

For Client Manager "Server Configuration - Details" option "CICS Socket Listener":
When "CICS Socket Listener" is enabled the iefcmn.srv file contains "IP_PERSISTCONN = N" i.e. non-persistent connection.
When "CICS Socket Listener" is not enabled the iefcmn.srv file contains "IP_PERSISTCONN = Y" i.e. persistent connection.

Customer confirms that "P_PERSISTCONN = Y" is set in the iefcmn.srv file i.e. "CICS Socket Listener" is not enabled.
It is recommended to enable because the server will close the socket after sending the response - see z/OS CICS Socket Listeners sections "Connection Life Time" and "z/OS CICS TCP/IP". 

For a successful client-server transaction request/response it will not actually matter if "CICS Socket Listener" is enabled or not because as long as the TIML or Gen server sends a response, the Client Manager (CM) will check the flag in the response buffer that tells it this is a transient (non-persistent) connection and it will close the connection.
However, if the transaction fails and there is no response e.g. Gen server crashes before sending a response, a network problem etc, then by not having "CICS Socket Listener" enabled the CM will keep the connection open for future use. That results in the CM trying to reuse a connection which does not exist at the server, and which will always result in an error. 

Check for any maintenance (network or otherwise) going on outside of normal business hours. When this is the only time the problem is seen, that could be the root cause. The fact that the client becomes slow would tend to indicate either a problem with the network or some other reason for TIML/Gen server lack of response.
With the current information available, it is not possible to provide any other root cause suggestions. To research further would require having a Client Manager log file (ideally with "Logging Level=Tracing") and a CICS log file both from the same timeframe when the problem occurs.

The suggested plan is:

• Change the CM to have the "CICS Socket Listener" option enabled i.e. "P_PERSISTCONN = N", that is the recommended setting. Per the explanation above, that will minimize the chance of client-side errors when transaction failures with no response occur.
  
  
• If the problem comes back after enabling that option then a new support case can be created at that time and the above logs provided for further research.