OneClick server is not able to establish CORBA communication to SpectroSERVER on port 14002 due to MTU Size
search cancel

OneClick server is not able to establish CORBA communication to SpectroSERVER on port 14002 due to MTU Size

book

Article ID: 216488

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

OneClick server is not able to establish CORBA communication to SpectroSERVER on port 14002 if the MTU configured on the SpectroSERVER or OneClick host system is too big for the underlying network path between these two hosts.

When tomcat is started, it will hang during startup at this point in the logging process:
 
(main) (GAPM) - GAPM.user group count : 0
(main) (GAPM) - GAPM.users count : 1
(main) (GAPM) - GAPM.finished processing user watch registration in : 232 ms
(main) (ALARMS) - Alarm fire timer scheduled at: 10000
(main) (ALARMS) - alarm.web.model.gatherpoolsize : 5
 
Netstat will show the connection established on both ends. Wireshark will show communication between the two servers as well.
 
Enabling CORBA debug reveals that OneClick is getting a connection refused exception when it tries to contact the CsCModelDomain service for login with the user model:
 
org.omg.CORBA.TRANSIENT: java.net.ConnectException: Connection refused (Connection refused)  vmcid: 0x0  minor code: 0  completed: No
at com.inprise.vbroker.IIOP.Connection.<init>(Unknown Source)
at com.inprise.vbroker.IIOP.Connection.<init>(Unknown Source)
at com.inprise.vbroker.IIOP.ConnectionFactory.create(Unknown Source)
at com.inprise.vbroker.IIOP.Bidder$Bid.connect(Unknown Source)
at com.inprise.vbroker.IIOP.Bidder$Bid.connect(Unknown Source)
at com.inprise.vbroker.orb.ConnectionPool.connect(Unknown Source)
at com.inprise.vbroker.orb.ConnectionPool.connect(Unknown Source)
at com.inprise.vbroker.GIOP.ProtocolConnector.reconnect(Unknown Source)
at com.inprise.vbroker.GIOP.ProtocolConnector.<init>(Unknown Source)
at com.inprise.vbroker.IIOP.ProtocolConnector.<init>(Unknown Source)
at com.inprise.vbroker.IIOP.Bidder$Bid.getConnector(Unknown Source)
at com.inprise.vbroker.IIOP.Bidder$Bid.getConnector(Unknown Source)
at com.inprise.vbroker.ProtocolEngine.PortfolioImpl.getConnector(Unknown Source)
at com.inprise.vbroker.ProtocolEngine.ManagerImpl.getConnector(Unknown Source)
at com.inprise.vbroker.orb.DelegateImpl._bind(Unknown Source)
at com.inprise.vbroker.orb.DelegateImpl.verifyConnection(Unknown Source)
at com.inprise.vbroker.orb.DelegateImpl.is_a(Unknown Source)
at org.omg.CORBA.portable.ObjectImpl._is_a(ObjectImpl.java:130)
at com.aprisma.spectrum.core.idl.CsCModelDomainHelper.narrow(CsCModelDomainHelper.java:33)
at com.aprisma.spectrum.core.idl.CsCModelDomainHelper.narrow(CsCModelDomainHelper.java:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.aprisma.visibroker.VBrokerCORBAHelper.getObjectImplementation(VBrokerCORBAHelper.java:756)
at com.aprisma.visibroker.VBrokerCORBAHelper.getObjectImplementation(VBrokerCORBAHelper.java:623)
at com.aprisma.util.corba.CORBAObjectMonitor.connect(CORBAObjectMonitor.java:968)
at com.aprisma.util.corba.CORBAObjectMonitor.connect(CORBAObjectMonitor.java:937)
at com.aprisma.spectrum.app.web.domain.UserModelDomainList.connectToDomain(UserModelDomainList.java:682)
at com.aprisma.spectrum.app.web.domain.UserModelDomainList.findInitialDomain(UserModelDomainList.java:753)
at com.aprisma.spectrum.app.web.domain.UserModelDomainList.findInitialUserModel(UserModelDomainList.java:830)
at com.aprisma.spectrum.app.web.domain.UserModelDomainList.init(UserModelDomainList.java:349)
at com.aprisma.spectrum.app.web.domain.AbstractModelDomainList.start(AbstractModelDomainList.java:978)
at com.aprisma.spectrum.app.web.domain.AbstractSSORBDomainList.start(AbstractSSORBDomainList.java:1465)
at com.aprisma.spectrum.app.util.context.ContextFactory.threadSafeStartContextObject(ContextFactory.java:260)
at com.aprisma.spectrum.app.util.context.ContextFactory.getContextObject(ContextFactory.java:164)
at com.aprisma.spectrum.app.web.domain.AdminModelDomainHelper.getDomainList(AdminModelDomainHelper.java:54)
at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.init(ModelRepository.java:869)
at com.aprisma.spectrum.app.topo.common.repository.AbstractModelRepository.start(AbstractModelRepository.java:490)
at com.aprisma.spectrum.app.util.context.ContextFactory.threadSafeStartContextObject(ContextFactory.java:260)
at com.aprisma.spectrum.app.util.context.ContextFactory.getContextObject(ContextFactory.java:164)
at com.aprisma.spectrum.app.web.servlet.ClientPollServlet.init(ClientPollServlet.java:531)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1123)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1078)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:971)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4849)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1133)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1868)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1045)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:429)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1577)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:934)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1382)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1372)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:907)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:933)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:637)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
25 May 2021 16:01:12,394 [main]  Info VBJ-Application.orb_com.inprise.vbroker.orb.ORB@5132d5bf.vbroker.log.default.filter.orb - SystemException ocurred. forward count=0 iors repeat=_forwardIORlatest is null ()

Environment

Release: 20.2.x
Component: SPCCSS - SpectroSERVER Core

Cause

Network Configuration

Resolution

CORBA packets have the do not fragment flag enabled. If the MTU size is too large for the network path between the two systems, the packets will be dropped which will cause the communication issue.
 
The following ping commands can be used to troubleshoot this within the network:
 
Examples:

LINUX:

MTU Size Too Big, as MTU is 1500( default ).
 
ping -s 2000 -M do <HOSTNAME>
PING <HOSTNAME> (<IP>) 2000(2028) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
 
Max Size for 1500 MTU, this should be successful if there are no MTU issues on the path.
 
ping -s 1472 -M do <HOSTNAME>
PING <HOSTNAME> (<IP>) 1472(1500) bytes of data.
1480 bytes from <HOSTNAME> (<IP>): icmp_seq=1 ttl=63 time=0.149 ms
1480 bytes from <HOSTNAME> (<IP>): icmp_seq=2 ttl=63 time=0.216 ms

WINDOWS:

ping -f -l 1472 <HOSTNAME>
 
Pinging <HOSTNAME> [<IP>] with 1472 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
 
Ping statistics for <IP>:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
 
$ ping -f -l 1372 <HOSTNAME>
 
Pinging <HOSTNAME> [<IP>] with 1372 bytes of data:
Reply from <IP>: bytes=1372 time=111ms TTL=53
Reply from <IP>: bytes=1372 time=112ms TTL=53
Reply from <IP>: bytes=1372 time=111ms TTL=53
Reply from <IP>: bytes=1372 time=113ms TTL=53
 
Ping statistics for <IP>
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 111ms, Maximum = 113ms, Average = 111ms
 
You will need to reduce the max MTU size on the Spectrum systems until the network path is configured appropriately. By default, this is set to 1500 but if the network path is not configured for that size, it will need to be reduced.

Additional Information

There is also a separate tool that can used to troubleshoot these network issues called iPerf3. Below is more information:

https://iperf.fr/

Powered by Wolken Software