Data collector disconnects on 3.2

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration

Issue/Introduction

Data collectors disconnect from the data aggregator on CAPM 3.2

Environment

CAPM 3.2 only

Cause

Failures are due to full TCP buffers between DA and DC causing a deadlock where no application level messages can get through the system.

Resolution

The procedure at a high-level is:

 

1.       Modify DA broker file.

2.       Modify DC broker files x (each DC)

3.       Stop DA application, stop DA broker.

4.       Restart DA application

5.       Restart DC broker x (each DC ) (or whenever the next DC broker restart occurs, new connections will be used).

 

Once above steps are complete, check for the number of open connections on the DA to verify that there are now connections on ports 6161*, and verify that there are no backups on queues.

 

CA  picked the next 2 even ports after 61616 but any available ports could be used as long as they are consistent between DA and DC. The odd ports are used in the documented encryption option. If that is used, then follow the steps to make each port the next one up  (like the documented 61616 to 61617 ) with the SSL configuration. 

 

You can check that there is not a backup on any Queue locally by looking at the DC Broker Admin page, or using the following command line on the system in case the admin page port is blocked:

 

/opt/IMDataCollector/broker/apache-activemq-5.13.1/bin/activemq dstat --jmxurl service:jmx:rmi:///jndi/rmi://localhost:11099/jmxrmi | awk '$2 > 10 { print $1, $2, $3, $4; }'

 

The detailed instructions of modification are below.  The following changes should prevent the lockup between the DC and DA brokers:

 

1)      On DA, open up 2 new ports for use by ActiveMQ broker network.

a.        In the <DA_INSTALL>/ broker/apache-activemq-5.13.1/conf/activemq.xml, replace the transport connector section with the following:

        <transportConnectors>

            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>

            <transportConnector name="openwire_poll" uri="tcp://0.0.0.0:61618"/>

            <transportConnector name="openwire_irep" uri="tcp://0.0.0.0:61620"/>

        </transportConnectors>

 

b.       Restart the DA broker. This could introduce slight data loss of a few messages in flight. 

2)      On each DC, use the newly configured transports for specific traffic.

                a.  Make a master copy of the section below, making the change to replace the DA HOSTNAME with the correct DA name. This common text will be used on each DC.

b.       In the <DC_INSTALL>/ broker/apache-activemq-5.13.1/conf/activemq.xml, replace the network connector section with the following. 

        <networkConnectors>

            <!-- Connect to another broker using a static configuration -->

                <networkConnector name="da_manager" uri="static:(tcp://DA-HOSTNAME:61616)" duplex="true" suppressDuplicateTopicSubscriptions="false">

                   <excludedDestinations>

                        <queue physicalName="PRQ"/>

                        <queue physicalName="DADistItemRepositoryMgr"/>

                        <queue physicalName="DIM.>"/>

                        <queue physicalName="DIP-poll.>"/>

                        <queue physicalName="DIP-req.>"/>

                   </excludedDestinations>

                </networkConnector>

 

                <networkConnector name="da_manager-PRQ" uri="static:(tcp:// DA-HOSTNAME:61618)" duplex="true" suppressDuplicateTopicSubscriptions="false">

                  <dynamicallyIncludedDestinations>

                        <queue physicalName="PRQ"/>

                   </dynamicallyIncludedDestinations>{

                </networkConnector>

 

                <networkConnector name="da_manager-DIrep" uri="static:(tcp:// DA-HOSTNAME:61620)" duplex="true" suppressDuplicateTopicSubscriptions="false">

                  <dynamicallyIncludedDestinations>

                        <queue physicalName="DADistItemRepositoryMgr"/>

                        <queue physicalName="DIM.>"/>

                        <queue physicalName="DIP-poll.>"/>

                        <queue physicalName="DIP-req.>"/>

                   </dynamicallyIncludedDestinations>

                </networkConnector>

        </networkConnectors>

 

3) Stop the DA and DA Broker

4) Restart the DA (which will start the broker as well )

5) Stop the DC and DC Broker. Restart the DC ( which will start the broker as well )

 

Additional Information

This configuration is in place out of the box in 3.5