UIM websphere_mq probe unable to collect some metrics

Products

DX Unified Infrastructure Management (Nimsoft / UIM) CA Unified Infrastructure Management SaaS (Nimsoft / UIM) Unified Infrastructure Management for Mainframe

Issue/Introduction

websphere_mq monitoring was last working Dec 10th. After that date we were seeing the following errors in the websphere_mq.log.

Dec 14 09:03:14:905 [Data Collector - XXXXXX_xxxx, websphere_mq] SENDING_ALARM: Self-Monitoring Failures for 'XXXXXX_xxxx:QUEUE.1227': Data Collection (50 of 50 failed). See websphere_mq.log for more details
Dec 14 09:03:14:905 [Data Collector - XXXXXX_xxxx, websphere_mq] ===== Self-Monitoring Alarm Failures: 1 alarm(s) sent for resource: XXXXXX_xxxx =====
Note 1: These Self-Monitoring Alarm Failures are aggregated for an element.metric type per resource. Individual failure details or related exceptions should proceed this log entry.
Note 2: 'Monitor Correlation' failures occur when a monitor does not find it's specific element in the inventory, or no metric value is available for the element.
With static monitors and changing inventory, these are sometimes expected and may be transitory.
Note 3: The failure count of 'Data Collection' failures often correlate with Monitor Correlation failures.
When there are only 'Data Collection' failures, or when they exceed 'Monitor Correlation' failures, that usually indicates a problem in collecting that metric value.
Some metric values are only available with additional system administration.
Some metric values are only available for specific element types. For instance one type of storage might have a metric, while another does not.
Generally it is desirable to understand 'Data Collection' failures for desired metrics, and sometimes the probe needs to be tuned for them

and/or

Dec 14 11:32:41:709 [Data Collector - XXXXXX_xxxx, websphere_mq] Data point lookup failed for '"ESC:QUEUE:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE"."1227"', will attempt to use raw data value for display
Dec 14 11:32:41:709 [Data Collector - XXXXXX_xxxx, websphere_mq] Failed to collect data for monitor 'XXXXXX_xxxx/SYSTEM.PROTECTION.ERROR.QUEUE.Oldest Message Age'. Updated value will not be available.: Lookup failed for value of monitor '"ESC:QUEUE:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE"."1227"'. GUI will not display current value or severity. Will try again next polling cycle.
Dec 14 11:32:41:709 [Data Collector - XXXXXX_xxxx, websphere_mq] Failed to collect data for monitor 'XXXXXX_xxxx/SYSTEM.PROTECTION.ERROR.QUEUE.Oldest Message Age'. Updated value will not be available.: Lookup failed for value of monitor '"ESC:QUEUE:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE"."1227"'. GUI will not display current value or severity. Will try again next polling cycle.
Dec 14 11:32:42:693 [BulkSender Monitor, websphere_mq] Sent NimQoS QOS_QUEUE_OLDEST_MESSAGE_AGE D, source=XXX-ab-th-fffff-02, target=XXXXXX_xxxx/SYSTEM.PROTECTION.ERROR.QUEUE, measuretime=1639499561701, value=null, stdev=0.0 and received confirmation id ZH50566020-41314
Dec 14 11:32:42:699 [BulkSender Monitor, websphere_mq] Sent NimQoS QOS_QUEUE_CURRENT_DEPTH D, source=XXX-ab-th-fffff-02, target=XXXXXX_xxxx/SYSTEM.PROTECTION.ERROR.QUEUE, measuretime=1639499561700, value=0.0, stdev=0.0 and received confirmation id ZH50566020-41339
monitorName = XXXXXX_xxxx/SYSTEM.PROTECTION.ERROR.QUEUE.Oldest Message Age
monitorKey = "ESC:QUEUE:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE"."1227"
monitorName = XXXXXX_xxxx/SYSTEM.PROTECTION.ERROR.QUEUE.Current Depth
monitorKey = "ESC:QUEUE:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE"."3"
Dec 14 11:33:41:645 [Data Collector - XXXXXX_xxxx, websphere_mq] no value found for monitor "ESC:QUEUE:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE"."1227" on parent idsProbe:XXXXXX_XXXX::SYSTEM.PROTECTION.ERROR.QUEUE::QUEUE

Environment

Release : 20.3

Component : UIM - WEBSPHERE_MQ

- wesphere_mq probe version 2.21/2.22

Cause

- IBM WebSphere MQ MONQ attribute was not set to High

Resolution

IBM Websphere MQ: MONQ monitoring attribute had to be set to 'High.'

See techdocs:

websphere_mq AC Configuration

See section titled: "Enable Queue Monitors on Linux and Solaris" as well as all of the other websphere_mq monitoring prerequisites.

Enable Queue Monitors on Linux and Solaris

The default configuration of WebSphere MQ does not expose all monitoring data of the Message Queues to an external application. Therefore, when the websphere_mq probe runs a command to fetch monitoring data, WebSphere MQ does not return any value for the following monitors:

Oldest Message Age
On Queue Time
Time Since Last Put
Time Since Last Get

Set the MONQ property value of the message queue to LOW, MEDIUM, or HIGH to fetch the monitoring data. Broadcom recommends you to use HIGH, which allows the probe to collect monitoring data about the message queues in the subsequent interval.

Follow these steps:

Open the Linux or Solaris terminal.
Go to the installation directory of WebSphere MQ. The default location is /opt/mqm/bin
Go to the Queue Manager of the target message queue that you want to monitor. For example, if your Queue Manager name is QM1, you can run the following command:

./runmqsc QM1

Run the DISPLAY ALL command and verify if the MONQ property value is OFF.
Run the ALTER QMGR MONQ(HIGH) command to set the MONQ property value to HIGH for all message queues.

Alternatively, run the ALTER QLOCAL(MYQUEUE.Q) MONQ(HIGH) to set the MONQ property value to HIGH for the given message queue. The QLOCAL is the queue type and MYQUEUE is the queue name.

On the Windows system, use the WebSphere MQ Explorer for updating the MONQ

property value.

Additional Information

For further information please refer to:

https://www.ibm.com/docs/en/ibm-mq/9.1?topic=monitoring-attributes-that-control-real-time