Multicast / JDBC Ping Full Tests - Clarity
search cancel

Multicast / JDBC Ping Full Tests - Clarity

book

Article ID: 201202

calendar_today

Updated On:

Products

Clarity PPM On Premise Clarity PPM SaaS

Issue/Introduction

Multicast /JDBC Ping tests (full recommendations and best practices in getting Multicast and JDBC Ping to work with Clarity) 

Note: It is recommended to use JDBC ping instead of Multicast. Please refer to : Multicast not working - Use JDBC Ping as Best Practice

To rule out any environmental issues, please change the multicast URL, port and beacon port, on both servers to something else, restart and try again first before proceeding further

Pre-requisites (Valid for both Mullticast and JDBC Ping)

The following pre-requisites must be in place in order for multicast communication to succeed between the various members participating in the Clarity cluster:

  1. All members must share a common multicast address
  2. Each Clarity ENV, e.g. Dev, Test, Prod each have their own mcast address so e.g. (230.0.1.1 for Dev, 230.0.1.2 for Test, 230.0.1.3 for Prod) etc
  3. The address should be CLASS D (between 224.0.0.0 and 239.255.255.255)
  4. All members must share a common multicast port
  5. Check (on all members in the cluster) that the specified port is not used by any other process – otherwise you need to find an alternate that is commonly unused on all
  6. All members must share a common NSA password
  7. Run "admin password” to reset if needed
  8. All members must be in the same subnet (confirm with ipconfig (windows) or ifconfig (unix) – if not then a multicast bridge needs to be configured (non-trivial) or servers need reconfiguring/relocating.  Be sure to double check the IP netmask used on all servers.  
    • Note:  A single netmask misconfiguration while provisioning the server can result in problems later on (ex:  servers are on 192.168.1.0 and 192.168.2.0 and a netmask of 255.255.254.0 is used.  If someone accidentally keys in a 255.255.255.0 netmask and the servers are spread across the 1.0 and 2.0 subnets, there will be communication and routing issues).
  9. If the server has multiple IP addresses (NICs) then configure the Beacon to bind  to a single specific IP address (Note: this is not required to ‘enable’ multicast but if the Beacons are not configured correctly then that will prevent servers being visible in the NSA) – Stop, Remove, Add and Deploy the Beacon Service after making any changes.

Environment

Release : Any

Resolution

Test 1: Admin Tower (Valid for both Mullticast and JDBC Ping)

  1. Ensure the beacon services have started successfully on ALL servers in the cluster (“service start beacon” on all application servers, followed by “service status beacon” on each application server to confirm the beacon service has started and is staying up)
  2. On the server running the NSA, run the following command:

admin tower

> refresh

> list clients

You should then get an output similar to this:

NSA: Active Clients:

--------------------------

1) server1

2) server2

3) server3

…

 

In total you should see ALL servers participating in the cluster.  If all servers are NOT showing AND beacons have been started successfully on all servers then we have a problem.

 

Next run this command:

> trace on

You should then start to see packets of traffic being transferred between ALL servers in the Clarity cluster.  If packets of data are not being sent between ALL servers then we have a problem.

 

Test 2: Jgroups Test (Valid for both Mullticast and JDBC Ping)

Needs beacon to be started

Jgroups is used for internal messaging by the application.  Jgroups uses multicast.  If there are network problems then Jgroups will not function correctly – this will cause problems in the application.  You can use the following tests to verify if Jgroups is able to successfully send messages between application servers.   This test can be used to identify if there are network/OS configuration issues which might be causing problems for the Jgroups component.

First stop all services (repeat for all application servers):

service stop all

Make sure all Services are stopped with command:

service status all

On the machine running NSA execute the following (SENDER). 

Note:  Replace <IPADDRESS> with the valid primary IP address bound to the system you are testing on.  On servers with more than one active NIC, be sure to use the IP address from the same IP subnet on all the servers when you conduct your test.

For UNIX:

export CLASSPATH=$NIKU_HOME/lib/jgroups-all.jar

For Windows:

set CLASSPATH=%NIKU_HOME%/lib/jgroups-all.jar

Then:

java org.jgroups.tests.McastSenderTest  -mcast_addr <multicast_address> -port <mcast port> -bind_addr <IPADDRESS>

e.g. “java org.jgroups.tests.McastSenderTest  -mcast_addr 239.255.252.3 -port 9090 –bind_addr <IPADDRESS>”

On of the app servers (RECEIVER):

Note:  Replace <IPADDRESS> with the valid primary IP address bound to the system you are testing on.  On servers with more than one active NIC, be sure to use the IP address from the same IP subnet on all the servers when you conduct your test.

For UNIX:

export CLASSPATH=$NIKU_HOME/lib/jgroups-all.jar

For Windows:

set CLASSPATH=%NIKU_HOME%/lib/jgroups-all.jar

Then :

java org.jgroups.tests.McastReceiverTest  -mcast_addr <multicast_address> -port <mcast port> -bind_addr <IPADDRESS>

e.g. java org.jgroups.tests.McastReceiverTest  -mcast_addr 239.255.252.3 -port 9090 –bind_addr <IPADDRESS>

 

NB: The same multicast address must be used on both the SENDER and on the RECEIVER.  You should use the same multicast address/port that you specify in the NSA.  If this fails you can try a different address/port and see if that succeeds.

 

Once this is running, you should be able to type a message in the terminal window of the “SENDER” – the same message should IMMEDIATELY appear in the window of the RECEIVER.  If this does not happen then we have a problem. This test should be done on each application server to see if it can RECEIVE messages from the SENDER.

Text should appear clearly without any “junk” being sent.  If text other than what has been sent from the SENDER is being seen in the RECEIVER terminals then this requires further investigation.

It is also recommended that you take screen shots by arranging 4 putty terminal windows on one screen with the Sender in the top left.  You then run the Receiver tests in the other sessions and you will be able to see simultaneously, if typed messages from the Sender terminal are appearing in the various Receiver terminals.  Please type in a clear message such as “HELLO WORLD!!” and not a small character like “/” so it can be easily seen in the terminal windows.

 

Troubleshooting the Jgroups Test

If you see this error:

java.net.BindException: Cannot assign requested address exception

Then add the “-Djava.net.preferIPv4Stack=true” property, e.g:

e.g.  java -Djava.net.preferIPv4Stack=true  org.jgroups.tests.McastReceiverTest -mcast_addr 239.255.252.3 -port 9090 –bind_addr 10.1.2.4

For more information on the diagnostic tests see: http://www.jgroups.org/manual/html/ch02.html

Test 3: Multicast Range(Valid for MULTICAST)

The other thing to note with multicast is there are a few ranges that Cisco advises not to use:

  x.0.0.x or x.128.0.x addresses (i.e., don't use 230.0.0.1).

We would also recommend using addresses in the range of 239.0.0.0/8 as these has been designated by RFC 2365 as a locally administered address space with local or organizational scope.  Please check what the multicast configuration is – if it falls into the above range then we have a problem.

 

Test 4: Network Interface (MULTICAST ONLY)

Running the command “ifconfig” should show “MULTICAST” on the network interfaces.  Lack of presence of “MULTICAST” flag indicates a problem.

 

Test 5: IGMP Snooping on Cisco Catalyst switches (MULTICAST ONLY)

If you are using Cisco network switches then you need to ensure IGMP snooping is enabled and an IGMP querier is also enabled for the VLAN – Your Network Team can confirm this.  Without the IGMP querier, the Cisco switch will not maintain the IGMP membership list and know which ports to deliver multicast traffic to. To enable it you can normally do this via the following command on the switch.  This is a VLAN level setting:

ip igmp snooping

igmp snooping querier

 

IGMP SNOOPING

IGMP Snooping is a layer 2 optimization for the layer 3 IGMP protocol.  This capability is present on all Cisco Catalyst switches.  IGMP Snooping is implemented on a specified VLAN and allows the switch to only forward multicast to the links that have solicited them.

IGMP QUERIER

The tables created for snooping (holding the member ports for each multicast group) are associated with the querier.  Without a querier the tables are not created and snooping WILL NOT WORK. 

 

Test 6: NSA Passwords on servers(Valid for both Mullticast and JDBC Ping)

Ensure that all of your NSA passwords on ALL servers are set to the same value.  This password is used to validate multicast packets sent by the various cluster nodes.  If these are not the same then we have a problem and we need to reset them all to the same value.  Run “admin password” to reset the password in the .passwd file.

 

Test 7: VMWare Servers (MULTICAST ONLY)

Promiscuous mode is NOT REQUIRED.

In a prior version of VMWare ESX, it was required that Promiscuous mode was needed in order to make multicast work correctly between VMs on an ESX cluster.  This is no longer the case on modern versions of VMware ESX.  Multicast should automatically work correctly inside VMware virtual networks.

Test 8: Bind Order of Network Interfaces (MULTICAST ONLY) 

It has been observed that in the case of Windows, that the bind order of network interfaces can affect the Beacon services.  In one instance in the field, a client had the MS LOOPBACK adaptor as first in the list – this caused issues for the Beacons and the issue was rectified after amending the order.  

On some Windows versions, such as Windows Server 2016 and Windows 10, you can use the interface metric to configure the order of network interfaces.

This is different than in previous versions of Windows and Windows Server, which allowed you to configure the binding order of network adapters by using either the user interface or the commands INetCfgComponentBindings::MoveBefore and INetCfgComponentBindings::MoveAfter. These two methods for ordering network interfaces are not available in Windows Server 2016 and Windows 10.

Instead, you can use the new method for setting the enumerated order of network adapters by configuring the interface metric of each adapter. You can configure the interface metric by using the Set-NetIPInterface Windows PowerShell command.

Powered by Wolken Software