This message is frequently seen in Collectors: "Timed out adding to outgoing message queue. Limit of <XXXX> reached." and on MOM this is seen: "com.wily.isengard.message.MessageUndeliverableEx"
Article ID: 9606
Updated On:
Products
Issue/Introduction
Frequently this message is seen in Collectors, where XXXX generally shows a number anywhere from 3000 and up.
"Timed out adding to outgoing message queue. Limit of XXXX reached"
Which is followed by this message on the MOM:
com.wily.isengard.message.MessageUndeliverableException: Outgoing mailbox is closed. Message cannot be sent
Other symptoms can include:
-Agents disconnecting and moving to other Collectors
-MOM disconnecting Collectors
-Workstation and/or Webview running slow
The above generally points to a performance issue. But we will concentrate more on the outgoing message queue.
The Enterprise Manager uses a "message queue" to communicate back and forth between EM's. So when you encounter a "Timed out adding to outgoing message queue. Limit of XXXX reached" message, this means that the message queue for the EM's communication protocol has filled up and no new messages will be added. This causes the Collector to cease sending out messages and all queries from the MOM to that Collector will error out. Outgoing delivery threads are fixed in number and shared by all the clients. These threads deliver messages from the outgoing message queue to the client, and only when there are no outgoing messages left in current queue move on to the next queue.
Some mail threads may be executing a large number of queries marking them busy on the connected Collectors. In this situation, the mail threads must wait for locks that are required to execute the requests. Since this is also blocked for a response until it times out, all of the mail threads will slow down as a result.
To alleviate this situation, there are 3 properties that work together.
transport.outgoingMessageQueueSize
This property is a hidden property inside the IntroscopeEnterpriseManger.properties file. This sets the size of the messaging queue. This property is based on the number of messages, not the size of the message content. If you increase this property value, you should also increase the heap size as the higher the number, the more RAM it uses. Basically the higher the value you use, the more resources it takes up.
transport.override.isengard.high.concurrency.pool.min.size
This property determines the minimum number of available outgoing delivery threads. We recommend to set the minimum and maximum the same. By doing this, we eliminate the adding/removing of pools when needed. These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottleneck for performance. However, as in the outgoing setting above, the higher the value, the more resources are taken up.
transport.override.isengard.high.concurrency.pool.max.size
This property determines the maximum number of available outgoing delivery threads. By increasing the maximum pool size, this allows more worker threads to send outgoing messages concurrently. These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottle neck for performance. However, as in the outgoing setting above, the higher the value, the more resources taken up.
The transport.override.isengard.high.concurrency.pool.max.size property in the IntroscopeEnterpriseManager properties file determines the maximum number of available outgoing delivery threads.
In a healthy cluster, the delivery threads run smoothly and it doesn't take long for a filled outgoing message queue to have space freed up for more messages. Delivery threads take messages off the outgoing message queues. If there are fewer delivery threads than active client connections, outgoing messages must wait in the queue, and the queue can reach its capacity. When there is no free space in the outgoing message queue, the Enterprise Manager cannot send requested data back to the client. This can result in serious Enterprise Manager performance problems. Increasing the queue size and thread pool size provides more capacity and can resolve this situation. These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottleneck for performance, but care must be taken when making the adjustments.
Environment
Application Performance Management 10.7 release.
Resolution
The following are the defaults for 9.x and 10.x versions of the Enterprise Manager, if not specified in the IntroscopeEnterpriseManager.properties file or commented out:
transport.outgoingMessageQueueSize=3000
transport.override.isengard.high.concurrency.pool.min.size=5
transport.override.isengard.high.concurrency.pool.max.size=5
These settings are considered low for the current and most recent versions and/or if your cluster has a very heavy load.
The recommended settings are below.
transport.outgoingMessageQueueSize=8000
transport.override.isengard.high.concurrency.pool.min.size=14
transport.override.isengard.high.concurrency.pool.max.size=14
The settings above can be increased but keep in mind the higher you increase them to, the more resources you will take up and can wind up choking the EM. Applying any changes above requires a restart of the EM.
NOTE: These settings should be applied to MOM and all Collectors.
If, after applying the above settings, you do not see any improvement, then there is more you can check on for your performance issue. See links below.
Note: Please keep in mind that you may encounter different recommended values. This is because, over time with Introscope, and as clusters grow, the recommended values need to be increased.
Additional Information
Feedback
