CA Gen 8.6 MQ Java Proxy handling of MQ timeout reason code 2033 (RC2033): MQRC_NO_MSG

Products

Gen Gen - Run Time Distributed

Issue/Introduction

Following the upgrade of CA Gen 7.0 MQ Java Proxy to 8.6, as part of a critical incident, it was analysed that when the transaction timed out after the specified MQ timeout, the established connection was not closed and resulted in an increase in the number of open connections between the application and MQ manager.
The MQ exception message was MQRC_NO_MSG_AVAILABLE, reason code 2033 (RC2033)
The MQ timeout is set in method getReplyTimeout() inside user exit MQSDynamicCoopFlowExit.java.
Have there been any changes in the handling of MQ exceptions between Gen 7.0 and 8.6 that might cause the connection not to be closed after the timeout?

Environment

Release : 8.6
Component : CA Gen Run Time, Proxies

Resolution

Instead of using the Gen 8.6 version of user exit MQSDynamicCoopFlowExit.java, the Gen 7.0 version of the exit with required custom code had been implemented with the Gen 8.6 java runtime by removing the "70" java runtime package name version references. NOTE: In 8.6 java runtime package name versioning has been completely removed.
However, compared to the 7.0 version, the 8.6 version of MQSDynamicCoopFlowExit.java has an extra method processMQException to handle the retry of failed MQ transactions/connections.

Gen java runtime MQSDynamicCoopflow.class checks for an MQ Exception on a Put or Get message and if one occurs a CSUException should then be thrown (that is the same behaviour in both 7.0 and 8.6 versions). In the 8.6 version if a CSUException occurs then the new method processMQException will be called.
Therefore, for MQ Exception message MQRC_NO_MSG_AVAILABLE (RC2033), a CSUException should have been thrown and then on the attempt to call the method processMQException a NoSuchMethodError exception would have been expected to be thrown because that method was not defined in the user exit. That would have caused the Java Proxy to have ended abnormally, so the open MQ connection was likely not closed correctly resulting in an increase of open connections.

Using prior versions of user exits is not a supported activity but rather the advice is to implement the Gen 8.6 version of user exit MQSDynamicCoopFlowExit.java which uses processMQException and move across the custom code from the 7.0 version. Then customise processMQException to also check for MQRC_NO_MSG_AVAILABLE to enable a retry for it.
NOTE: There was also a pre-Gen 8.6 Complete fix made for Java MQ communication retries in PTF RTJ86103/RO99131 which updated the runtime file odc.mqs.jar. That same fix is in Gen 8.6 Complete PTF WKS86200/SO09618. Therefore, in order to pick up the above fix, after implementing the 8.6 version of the MQSDynamicCoopFlowExit.java user exit it should also be ensured that the Gen Java runtimes being used have been generated from a Gen 8.6 Complete (WKS86200) environment.

Additional Information

1. IBM MQ 9.1.x > IBM MQ > Troubleshooting and support > Reason codes and exceptions > API completion and reason codes > API reason codes

2. The Gen 8.6 MQ Java runtime error processing with retries operates as follows:
There is a pool of connection objects for connecting to queues in the MQ queue manager which initially has 20 objects but may be expanded.
Every time there is an error response the runtime closes the queues and then check for a retry by calling processMQException.
If a retry is allowed the code will look for an existing, unused connection in the pool and if one does not exist then a new connection will be created.
For an MQ timeout it does not mean that the connection is bad but rather the server just took too long to respond, so that is not a reason to delete the connection from the pool.
So for all retry attempts, the queues will be closed and then typically the same connection will be reused.
If all retry attempts fail then an error is returned to the client, and all connections to the queue in the queue manager will be closed.