Interface data is no longer seen in Dashboards in CA Performance Manager post r2.6 upgrade

Article Id: 38199
Status: Published
Updated On: 14-02-2018 07:36
Legacy Id: TEC1254378
Products:
CA Infrastructure Management CA Infrastructure Management CA Performance Management - Usage and Administration CA Performance Management - Data Polling
Issue/Introduction:

Problem:

Interface data is no longer present in Dashboard Views in CA Performance Manager reports post r2.6 upgrade.

After upgrading to the new r2.6 release, Interface data is missing from reports.

 

Cause:

The cause of this is a code defect introduced in the r2.6 release. The key error message that ties the missing Interface data occurrence to this known problem is found in the karaf.log file on the Data Aggregator server.

WARN | ssion Task-95854 | 2016-02-17 08:12:32,708 | ActiveMQMessageConsumer | activemq.ActiveMQMessageConsumer 1436 | e.activemq.activemq-osgi | | ID:andprdnetqos03.min.navisite-int.net-46557-1446205260086-3:5:1:1 suppressing duplicate delivery on connection, poison acking: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:andprdnetqos03.min.navisite-int.net-46557-1446205260086-3:5:1:1, destination = queue://PRQ, message = ActiveMQObjectMessage {commandId = -1875054001, responseRequired = true, messageId = ID:andprdnetqos04.min.navisite-int.net-54221-1446200433896-1:4:45:1:2188674977, originalDestination = null, originalTransactionId = TX:ID:andprdnetqos04.min.navisite-int.net-54221-1446200433896-1:4:309342512, producerId = da_broker->dc_broker_5390c032-d6e4-4731-a170-bd702e86d870-38718-1446046526271-8:2:1:1, destination = queue://PRQ, transactionId = null, expiration = 0, timestamp = 1455714752673, arrival = 0, brokerInTime = 1455714752684, brokerOutTime = 1455714752689, correlationId = 7651c601-4666-4f84-b7ab-ee5f9250f29e, replyTo = null, persistent = false, type = com.ca.im.dm.core.common.ProcessedPollResponse, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = true, userID = null, content = [email protected], marshalledProperties = [email protected], dataStructure = null, redeliveryCounter = 0, size = 0, properties = {ca.poi=true}, readOnlyProperties = true, readOnlyBody = true, droppable = false, jmsXGroupFirstForConsumer = false}, redeliveryCounter = 0} 

In a recent instance of this problem being reported, we observed over 45 thousand errors appear within a 4 minute period.

The karaf.log is found on the Data Aggregator in the following location:

$INSTALL_HOME/IMDataAggregator/apache-karaf-<version>/data/log/

The default installation path would be here under /opt:

/opt/IMDataAggregator/apache-karaf-<version>/data/log/

 

Solution:

  • As previously referenced this is a known defect within the r2.6 release code.
  • This is fixed in the GA r2.7 release of CA Performance Manager.
  • No workaround is available that would prevent this from occurring again without upgrading to the r2.7 release.
  • Until an upgrade is performed, the problem can be temporarily resolved by restarting the Data Collector(s) dcmd service that is tied to the Data Aggregator involved.
  • The problem is intermittent and is normally only seen against Data Collector(s) tied to the DA that have been running for a long time. As such, the problem may return and require another Data Collector dcmd service restart until the r2.7 upgrade is performed.

 

From the r2.7 Fixed Issues list found in the r2.7 online documentation Wiki, this is confirmed as resolved in r2.7 per this entry:

Reference Number: DE49736

Symptom: AMQ poison ack messages can overrun karaf.log on the Data Aggregator.

Resolution: ActiveMQConsumer output is written to a separate ActiveMQConsumer log on the Data Aggregator.

The full Fixed Issues list can found here or here at this full URL:

https://docops.ca.com/ca-performance-management/2-7/en/release-notes/fixed-issues

Environment:

Release: IMDAGG99000-2.6-Infrastructure Management-Data Aggregator
Component: