Gen TISRVMSL multiple socket errors occur under z/OS 2.5 or higher version

Products

• Gen
• Gen - Run Time Distributed

Issue/Introduction

After upgrading to z/OS 2.5 seeing problems with the Gen Multi Sockets CICS Listener TISRVMSL and its error handling.
After some time since the TISRVMSL was started, these symptoms may start to appear:

The server-side CICS MSGUSR log will contain many messages of the form: 
*****
TISRVMSL TIML TASK=######## MM/DD/YYYY HH:MM:SS START TRANSACTION FAILED FOR TRAN Gen_Server_Trancode1
TISRVMSL TIML TASK=######## MM/DD/YYYY HH:MM:SS CICS ERROR, RESPONSE=00000069, RESP2=00000008
TISRVMSL TIML TASK=######## MM/DD/YYYY HH:MM:SS LISTENER TAKESOCKET BACK FAILED FOR SERVER Gen_Server_Trancode1
TISRVMSL TIML TASK=######## MM/DD/YYYY HH:MM:SS SOCKET ERROR, RETCODE=-0000001, ERRNO=00001002
TISRVMSL TIML TASK=######## MM/DD/YYYY HH:MM:SS ACCEPT CALL FAILED
TISRVMSL TIML TASK=######## MM/DD/YYYY HH:MM:SS SOCKET ERROR, RETCODE=-0000001, ERRNO=00001002
*****

NOTE: RESPONSE=00000069 means USERIDERR and RESP2=00000008 means the USERID is unknown,  thus the above error situation is for a security failure. 

Similar errors also reported after upgrade to z/OS 3.1 which caused a CICS region to lock up:
*****
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS CICS INQUIRE TRANSID NOT ENABLED FOR SERVER TRANSACTION Gen_Server_Trancode1
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS CICS ERROR, RESPONSE=00000000, RESP2=00000000
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS CICS INQUIRE TRANSID NOT ENABLED FOR SERVER TRANSACTION Gen_Server_Trancode2
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS CICS ERROR, RESPONSE=00000000, RESP2=00000000
...
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS ACCEPT CALL FAILED
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS SOCKET ERROR, RETCODE=-0000001, ERRNO=00001002
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS ACCEPT CALL FAILED
.TISRVMSL TISM TASK=######## MM/DD/YYYY HH:MM:SS SOCKET ERROR, RETCODE=-0000001, ERRNO=00001002                    
*****

Client connections may also hang/loop/spin when a security violation occurs instead of returning the expected message "TIRM621E: Error creating semaphore, can't access server"
Also, the MSGUSR log does not contain the expected message "GIVESOCKET NOT TAKEN".
(Gen EDGE Community post: Security violations behavior has changed in GUI client-server)

Environment

Gen 8.6 COBOL/CICS Distributed Process servers using TISRVMSL under z/OS 2.5 or higher version.

Cause

There is a known TISRVMSL problem that only seems to get exposed with TCP/IP changes under z/OS 2.5 or higher version.
The TISRVMSL error handling of conditions like security violations/failures does not close the current socket and eventually the available socket pool is exhausted which results in the "ACCEPT CALL FAILED" messages.

Resolution

Workaround

Recycle the TISRVMSL 

Resolution

Apply Gen 8.6 PTF LU08295 ("TISRVMSL LOOPS/DOES NOT PROCESS SOCKETS WHEN HANDLING ERRORS") 
NOTE: This PTF only affects TISRVMSL, as it fixes a problem with socket management handling when sending the error response. It does not impact the actual Gen servers so the PTF should be relatively easy to deploy.

When updating the TISRVMSL in required CICS region:

1. Stop the TISRVMSL listener with CICS command: EZAO,STOP,LISTENER(TRANSID)
   NOTE: The TRANSID is the transaction identifier defined for the TISRVMSL in the EZACONFG file and CICS definition per the Gen doc. page Communications and Middleware. So the value may be TIML, but it may also have been defined differently.
   The TRANSID can also be found using command: CEMT I TRAN(*) PRO(TISRVMSL)
   
   
2. Deploy the new version of TISRVMSL from SMP/E target to the correct CICS DFHRPL library (Librarydsn) and NEWCOPY it.
   
   
3. Start the TISRVMSL listener with CICS command: EZAO,START,LISTENER(TRANSID)

Additional Information

If in doubt whether the PTF has already been applied the current TISRVMSL version can be checked as follows:

1. Locate the relevant CICS DFHRPL library using the CICS command: CEMT I PROG(TISRVMSL)
   Put cursor beside the Prog(TISRVMSL) and hit enter; scroll and find Librarydsn  e.g.
   
   
   
   
   
   
2. In Librarydsn locate the TISRVMSL load member and browse it. Look for string TISRVMSL(U) and check the date after it. If PTF LU08295 (or later PTF containing the same fix) is applied the date should be 11/23/22 or later. For example: