Multicast /JDBC Ping tests (full recommendations and best practices in getting Multicast and JDBC Ping to work with Clarity)
Note: It is recommended to use JDBC ping instead of Multicast. Please refer to : Multicast not working - Use JDBC Ping as Best Practice
To rule out any environmental issues, please change the multicast URL, port and beacon port, on both servers to something else, restart and try again first before proceeding further
Pre-requisites (Valid for both Mullticast and JDBC Ping)
The following pre-requisites must be in place in order for multicast communication to succeed between the various members participating in the Clarity cluster:
Release : Any
Test 1: Admin Tower (Valid for both Mullticast and JDBC Ping)
admin tower
> refresh
> list clients
You should then get an output similar to this:
NSA: Active Clients:
--------------------------
1) server1
2) server2
3) server3
…
In total you should see ALL servers participating in the cluster. If all servers are NOT showing AND beacons have been started successfully on all servers then we have a problem.
Next run this command:
> trace on
You should then start to see packets of traffic being transferred between ALL servers in the Clarity cluster. If packets of data are not being sent between ALL servers then we have a problem.
Test 2: Jgroups Test (Valid for both Mullticast and JDBC Ping)
Needs beacon to be started
Jgroups is used for internal messaging by the application. Jgroups uses multicast. If there are network problems then Jgroups will not function correctly – this will cause problems in the application. You can use the following tests to verify if Jgroups is able to successfully send messages between application servers. This test can be used to identify if there are network/OS configuration issues which might be causing problems for the Jgroups component.
First stop all services (repeat for all application servers):
service stop all
Make sure all Services are stopped with command:
service status all
On the machine running NSA execute the following (SENDER).
Note: Replace <IPADDRESS> with the valid primary IP address bound to the system you are testing on. On servers with more than one active NIC, be sure to use the IP address from the same IP subnet on all the servers when you conduct your test.
For UNIX:
export CLASSPATH=$NIKU_HOME/lib/jgroups-all.jar
For Windows:
set CLASSPATH=%NIKU_HOME%/lib/jgroups-all.jar
Then:
java org.jgroups.tests.McastSenderTest -mcast_addr <multicast_address> -port <mcast port> -bind_addr <IPADDRESS>
e.g. “java org.jgroups.tests.McastSenderTest -mcast_addr 239.255.252.3 -port 9090 –bind_addr <IPADDRESS>”
On of the app servers (RECEIVER):
Note: Replace <IPADDRESS> with the valid primary IP address bound to the system you are testing on. On servers with more than one active NIC, be sure to use the IP address from the same IP subnet on all the servers when you conduct your test.
For UNIX:
export CLASSPATH=$NIKU_HOME/lib/jgroups-all.jar
For Windows:
set CLASSPATH=%NIKU_HOME%/lib/jgroups-all.jar
Then :
java org.jgroups.tests.McastReceiverTest -mcast_addr <multicast_address> -port <mcast port> -bind_addr <IPADDRESS>
e.g. java org.jgroups.tests.McastReceiverTest -mcast_addr 239.255.252.3 -port 9090 –bind_addr <IPADDRESS>
NB: The same multicast address must be used on both the SENDER and on the RECEIVER. You should use the same multicast address/port that you specify in the NSA. If this fails you can try a different address/port and see if that succeeds.
Once this is running, you should be able to type a message in the terminal window of the “SENDER” – the same message should IMMEDIATELY appear in the window of the RECEIVER. If this does not happen then we have a problem. This test should be done on each application server to see if it can RECEIVE messages from the SENDER.
Text should appear clearly without any “junk” being sent. If text other than what has been sent from the SENDER is being seen in the RECEIVER terminals then this requires further investigation.
It is also recommended that you take screen shots by arranging 4 putty terminal windows on one screen with the Sender in the top left. You then run the Receiver tests in the other sessions and you will be able to see simultaneously, if typed messages from the Sender terminal are appearing in the various Receiver terminals. Please type in a clear message such as “HELLO WORLD!!” and not a small character like “/” so it can be easily seen in the terminal windows.
Troubleshooting the Jgroups Test
If you see this error:
java.net.BindException: Cannot assign requested address exception
Then add the “-Djava.net.preferIPv4Stack=true” property, e.g:
e.g. java -Djava.net.preferIPv4Stack=true org.jgroups.tests.McastReceiverTest -mcast_addr 239.255.252.3 -port 9090 –bind_addr 10.1.2.4
For more information on the diagnostic tests see: http://www.jgroups.org/manual/html/ch02.html
Test 3: Multicast Range(Valid for MULTICAST)
The other thing to note with multicast is there are a few ranges that Cisco advises not to use:
x.0.0.x or x.128.0.x addresses (i.e., don't use 230.0.0.1).
We would also recommend using addresses in the range of 239.0.0.0/8 as these has been designated by RFC 2365 as a locally administered address space with local or organizational scope. Please check what the multicast configuration is – if it falls into the above range then we have a problem.
Test 4: Network Interface (MULTICAST ONLY)
Running the command “ifconfig” should show “MULTICAST” on the network interfaces. Lack of presence of “MULTICAST” flag indicates a problem.
Test 5: IGMP Snooping on Cisco Catalyst switches (MULTICAST ONLY)
If you are using Cisco network switches then you need to ensure IGMP snooping is enabled and an IGMP querier is also enabled for the VLAN – Your Network Team can confirm this. Without the IGMP querier, the Cisco switch will not maintain the IGMP membership list and know which ports to deliver multicast traffic to. To enable it you can normally do this via the following command on the switch. This is a VLAN level setting:
ip igmp snooping
igmp snooping querier
IGMP SNOOPING
IGMP Snooping is a layer 2 optimization for the layer 3 IGMP protocol. This capability is present on all Cisco Catalyst switches. IGMP Snooping is implemented on a specified VLAN and allows the switch to only forward multicast to the links that have solicited them.
IGMP QUERIER
The tables created for snooping (holding the member ports for each multicast group) are associated with the querier. Without a querier the tables are not created and snooping WILL NOT WORK.
Test 6: NSA Passwords on servers(Valid for both Mullticast and JDBC Ping)
Ensure that all of your NSA passwords on ALL servers are set to the same value. This password is used to validate multicast packets sent by the various cluster nodes. If these are not the same then we have a problem and we need to reset them all to the same value. Run “admin password” to reset the password in the .passwd file.
Test 7: VMWare Servers (MULTICAST ONLY)
Promiscuous mode is NOT REQUIRED.
In a prior version of VMWare ESX, it was required that Promiscuous mode was needed in order to make multicast work correctly between VMs on an ESX cluster. This is no longer the case on modern versions of VMware ESX. Multicast should automatically work correctly inside VMware virtual networks.
Test 8: Bind Order of Network Interfaces (MULTICAST ONLY)
It has been observed that in the case of Windows, that the bind order of network interfaces can affect the Beacon services. In one instance in the field, a client had the MS LOOPBACK adaptor as first in the list – this caused issues for the Beacons and the issue was rectified after amending the order.
On some Windows versions, such as Windows Server 2016 and Windows 10, you can use the interface metric to configure the order of network interfaces.
This is different than in previous versions of Windows and Windows Server, which allowed you to configure the binding order of network adapters by using either the user interface or the commands INetCfgComponentBindings::MoveBefore and INetCfgComponentBindings::MoveAfter. These two methods for ordering network interfaces are not available in Windows Server 2016 and Windows 10.
Instead, you can use the new method for setting the enumerated order of network adapters by configuring the interface metric of each adapter. You can configure the interface metric by using the Set-NetIPInterface Windows PowerShell command.