Increase in Gen .NET proxy to CICS (TISRVMSL) TCP socket exceptions

Products

• Gen
• Gen - Run Time Distributed

Issue/Introduction

Since early April 2024 the error rate for a Gen 7.6 .NET proxy application connected to Gen 8.6 CICS servers via Gen CICS Multi-Sockets server Listener (TISRVMSL / TIML) has increased five-fold across the board.  For some transactions this has caused a significant backlog of manual remediation.

It is appreciated that Gen 7.6 has ended support since 2012, but seeking assistance to diagnose (and hopefully fix) the cause of the increase.

The 2 most common errors encountered, accounting for 95+%, are:
54.3 % were caused by failed service calls to CoopFlow calls from proxies with com.ca.gen.csu.exception.GenException
41.3 % were caused by failed service calls to CoopFlow calls from proxies to with com.ca.gen.odc.OdcCommException

Specific details
com.ca.gen.csu.exception.GenException
5,760    failed requests
[Function: TCPIPConnection::readData]System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 size, SocketFlags socketFlags) at com.ca.gen.odc.
5,120    failed requests
[Function: TCPIPConnection::writeData]System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Sockets.Socket'. at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags,
192    failed requests
[Function: TCPIPConnection::writeData]System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Send(Byte[] buffer) at com.ca.gen.odc.coopflow.tcpip.TCPIPConnection.writeDat
128    failed requests
[Function: CoopFlow::PerformFlow]Caught unexpected exception: System.Threading.ThreadAbortException: Thread was being aborted. at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socket
128    failed requests
[Function: TCPIPConnection::writeData]System.Net.Sockets.SocketException: A request to send or receive data was disallowed because the socket had already been shut down in that direction with a previous shutdown call at System.Net.Sockets.Socket.
128    failed requests
[Function: TCPIPConnection::readData]System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Sockets.Socket'. at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags
64    failed requests

com.ca.gen.odc.OdcCommException
5,056    failed requests
[Function: CFBDynamicMessage::parseForCommError]Received peer communication error: 705 :
5,056    failed requests

Environment

Gen .NET Proxy client to CICS Multi Sockets Listener TISRVMSL 

Resolution

The TISRVMSL expects a non-persistent client connection.
Not setting the connection_persistence flag to N on a client will result in the listener closing the socket on first use and then potential socket reuse attempts (retries) on the client.
The high count failure reason exceptions like "An existing connection was forcibly closed by the remote host" do point to a persistent connection being used.
The 7.6 .NET Proxy does not have the option to set a connection_persistence flag in the commcfg.txt file because that only came in during Gen 8.x, so it would always use a persistent connection. So the client behaviour in response to a closed socket would depend on the retry logic and the default 7.6 . NET coopflow user exit TCPIPDynamicCoopFlowExit.cs has this retry logic with a limit of 2 connection attempts:
*****
            // check for reset by peer exception messages
            if (message.IndexOf("Connection reset by peer") != -1 ||
                 message.IndexOf("Connection unexpectantly closed by server") != -1)
*****
So that does not cover the exceptions reported like "An existing connection was forcibly closed by the remote host" .

It is believed that a customised user exit TCPIPDynamicCoopFlowExit.cs with these 2 extra message checks and also 5 connection attempts instead of 2 is being used in Production:
"An existing connection was forcibly closed by the remote host"
"An established connection was aborted by the software in your host machine"

Therefore it would seem that if those messages are already being checked in the user exit to now get them reported as runtime exceptions means the connection attempts in the exit must have started to exceed the 5 attempts.
As the reported runtime exceptions only started to increase since April 2024 with no change in the deployed Gen software, why the current number of 5 attempts is no longer always sufficient points to some non-Gen environment change and possibly network related.
Potentially increasing the number of retries in the exit may help but that is uncertain because it is not known why the initial connection attempts are failing and whether they are related to "short duration" or "long duration" connectivity problems.

For this particular scenario no confirmed resolutions was found at the time of writing this article.

Additional Information

.NET Proxy > Preparing for Execution