search cancel

Spectrum Oneclick tomcat is started but we are not able to login.

book

Article ID: 230436

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

Oneclick  server crashed. We restarted tomcat and can see that its started but we are not able to login to the console. 

We see the following in the logs 

org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-2052"]
13-Dec-2021 12:18:25.548 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-2052]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
Caused by: java.net.BindException: Address already in use

Environment

Release : 21.2

Component : Spectrum OneClick

Cause

There was a java hung process, this can happen if your run the stopTomcat.sh and immediately startTomcat.sh. The OS does not have enough time to free the port and the new process cannot bind the port.

In more recent tomcat releases, it takes a lot longer to let go off the tomcat processes after stopping.  this can be caused by a known tomcat issue where it does not release the java connection on the tomcat port after stopping tomcat so on restart tomcat cannot bind to the port as it's help by the previous instance of Java.  

Resolution

Stop Tomcat and webtomcat to ensure all Java processes are down. If not, then kill -9 the Java process that is still running, and then restart tomcat and webtomcat. 

i.e. after stopping tomcat, check the tomcat process using 

 ps -ef |grep -i UseG1GC

the grep process can be ignored

spectrum 23048 22987  0 12:08 pts/0    00:00:00 grep --color=auto -i UseG1GC

But if we see the tomcat/java process as follows, we need to kill it before starting tomcat again. 

"kill -9  1304" would kill the process below.


spectrum  1304     1  1 07:35 ?        00:03:02 /usr/Spectrum/Java/bin/java -Djava.util.logging.config.file=/usr/Spectrum/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DOneClick -server -Xmx16384M -Xms4096M -XX:+UseG1GC -XX:G1HeapRegionSize=8M -XX:G1HeapWastePercent=5 -XX:MaxGCPauseMillis=800 -XX:SurvivorRatio=8 -Dsun.net.inetaddr.ttl=60 -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djavax.net.ssl.trustStore=/usr/Spectrum/custom/keystore/cacerts -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Djava.endorsed.dirs=/usr/Spectrum/tomcat/lib/endorsed -classpath :/usr/Spectrum/tomcat/lib/mail.jar:/usr/Spectrum/tomcat/bin/bootstrap.jar:/usr/Spectrum/tomcat/bin/tomcat-juli.jar:/usr/Spectrum/tomcat/classes -Djava.security.manager -Djava.security.policy==/usr/Spectrum/tomcat/conf/catalina.policy -Dcatalina.base=/usr/Spectrum/tomcat -Dcatalina.home=/usr/Spectrum/tomcat -Djava.io.tmpdir=/usr/Spectrum/tomcat/temp -Djava.library.path=/opt/SPECTRUM/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib:/usr/Spectrum/tomcat/bin org.apache.catalina.startup.Bootstrap start