Data Aggregator shutting down or not starting
Article ID: 255121
Updated On:
Products
Issue/Introduction
At Data Aggregator & Data Collector server, the status of activemq is down when you test using 'service activemq status', but it’s running an activemq processes on both servers.
# service dadaemon status
/opt/IMDataAggregator/scripts/karaffunctions: line 178: -1: substring expression < 0
IM Data Aggregator is running, pid=10707
# service activemq status
Status for ActiveMQ: INFO: Loading '/opt/IMDataAggregator/broker/apache-activemq-5.16.2//bin/env'
INFO: Using java '/opt/IMDataAggregator/jre/bin/java'
ActiveMQ not running
# ps -ef | grep active
root 10546 1 1 Nov21 ? 00:18:15 /opt/IMDataAggregator/jre/bin/java -Xms1024M -Xmx2048M -Xmn596M -server -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:TargetSurvivorRatio=90 -XX:InitialTenuringThreshold=15 -XX:MaxTenuringThreshold=15 -XX:+ScavengeBeforeFullGC -XX:+ExplicitGCInvokesConcurrent -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError=/opt/IMDataAggregator/scripts/activemq stop -Dcom.sun.management.jmxremote.port=11099 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/conf/jmx.password -Dcom.sun.management.jmxremote.access.file=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/conf/jmx.access -Dcom.sun.management.jmxremote.password.toHashes=false -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/tmp --add-reads=java.xml=java.logging --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED -Dactivemq.classpath=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/conf:/opt/IMDataAggregator/broker/apache-activemq-5.16.2/../lib/: -Dactivemq.home=/opt/IMDataAggregator/broker/apache-activemq-5.16.2 -Dactivemq.base=/opt/IMDataAggregator/broker/apache-activemq-5.16.2 -Dactivemq.conf=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/conf -Dactivemq.data=/opt/IMDataAggregator/broker/apache-activemq-5.16.2/data -jar /opt/IMDataAggregator/broker/apache-activemq-5.16.2/bin/activemq.jar start
root 10707 10628 5 Nov21 ? 01:29:11 /opt/IMDataAggregator/jre/bin/java -Xms2048M -Xmx8192M -server -Xms2048M -Xmx8192M -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.toHashes=false -XX:NewSize=1535m -XX:NewRatio=3 -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:TargetSurvivorRatio=50 -XX:InitialTenuringThreshold=15 -XX:MaxTenuringThreshold=15 -XX:+ScavengeBeforeFullGC -XX:+ExplicitGCInvokesConcurrent -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=/opt/IMDataAggregator/apache-karaf-4.3.3/lib/endorsed/org.apache.karaf.specs.locator-4.3.3.jar --patch-module java.xml=/opt/IMDataAggregator/apache-karaf-4.3.3/lib/endorsed/org.apache.karaf.specs.java.xml-4.3.3.jar --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.file=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=java.base/sun.net.www.content.text=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.security.sasl/com.sun.security.sasl=ALL-UNNAMED -Dkaraf.instances=/opt/IMDataAggregator/apache-karaf-4.3.3/instances -Dkaraf.home=/opt/IMDataAggregator/apache-karaf-4.3.3 -Dkaraf.base=/opt/IMDataAggregator/apache-karaf-4.3.3 -Dkaraf.data=/opt/IMDataAggregator/apache-karaf-4.3.3/data -Dkaraf.etc=/opt/IMDataAggregator/apache-karaf-4.3.3/etc -Dda.data.home=/opt/IMDataAggregator/data -DACTIVEMQ_HOME=/opt/IMDataAggregator/broker/apache-activemq-5.16.2 -Dda.version=21.2.8.335 -Dkaraf.log=/opt/IMDataAggregator/apache-karaf-4.3.3/data/log -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/opt/IMDataAggregator/apache-karaf-4.3.3/data/tmp -Djava.util.logging.config.file=/opt/IMDataAggregator/apache-karaf-4.3.3/etc/java.util.logging.properties -XX:+HeapDumpOnOutOfMemoryError -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Djava.io.tmpdir=/opt/IMDataAggregator/apache-karaf-4.3.3/data/tmp -XX:OnOutOfMemoryError=/opt/IMDataAggregator/apache-karaf-4.3.3/bin/restart -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /opt/IMDataAggregator/apache-karaf-4.3.3/lib/karaf*.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/boot/org.apache.karaf.diagnostic.boot-4.3.3.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/boot/org.apache.karaf.jaas.boot-4.3.3.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/boot/org.apache.karaf.main-4.3.3.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/boot/org.apache.karaf.specs.activator-4.3.3.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/boot/osgi.core-7.0.0.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/jdk9plus/istack-commons-runtime-3.0.10.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/jdk9plus/jakarta.xml.bind-api-2.3.3.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/jdk9plus/javax.activation-1.2.0.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/jdk9plus/javax.annotation-api-1.3.1.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/jdk9plus/jaxb-runtime-2.3.3.jar:/opt/IMDataAggregator/apache-karaf-4.3.3/lib/jdk9plus/txw2-2.3.3.jar org.apache.karaf.main.Main
We can see in activemq.out that there is an exception due to port already in use:
# tail -f activemq.out
... 3 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.bind(Unknown Source)
at java.base/java.net.ServerSocket.bind(Unknown Source)
at java.base/java.net.ServerSocket.<init>(Unknown Source)
at java.base/java.net.ServerSocket.<init>(Unknown Source)
at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(Unknown Source)
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(Unknown Source)
... 12 more
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 11099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 11099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(Unknown Source)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Unknown Source)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Unknown Source)
Caused by: java.rmi.server.ExportException: Port already in use: 11099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(Unknown Source)
at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(Unknown Source)
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(Unknown Source)
at java.rmi/sun.rmi.transport.LiveRef.exportObject(Unknown Source)
at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(Unknown Source)
at java.rmi/sun.rmi.registry.RegistryImpl.setup(Unknown Source)
at java.rmi/sun.rmi.registry.RegistryImpl.<init>(Unknown Source)
at jdk.management.agent/sun.management.jmxremote.SingleEntryRegistry.<init>(Unknown Source)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(Unknown Source)
... 3 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.bind(Unknown Source)
at java.base/java.net.ServerSocket.bind(Unknown Source)
at java.base/java.net.ServerSocket.<init>(Unknown Source)
at java.base/java.net.ServerSocket.<init>(Unknown Source)
at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(Unknown Source)
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(Unknown Source)
... 12 more
Environment
Release : any
Resolution
1. service dadaemon stop, service activemq stop
2. cd /opt/IMDataAggregator/apache-karaf-x.y.z
3. mv data data.old
4. cd deploy
5. rm -f *.xml
6. service dadaemon start
Additional Information
Feedback
