mon:thread: ACE has timed out
search cancel

mon:thread: ACE has timed out

book

Article ID: 101638

calendar_today

Updated On:

Products

iDash Workload Automation

Issue/Introduction


The iXp users repeatedly seeing the following message:
Warning: this data is more than five(5) minutes old.
If this message persists, please contact your iXp administrator

 
$IXP_HOME/log/ixpDaemon.log shows the following messages: 
mon:thread: ACE has timed out 
mon:thread: ACE has timed out 
........
......
...
gc:iac:1:0:0/0:0:lac:0/0:0:jtc:0/0:0 
gc:bef=472M:avg=665M:aft=493M:low=94M:59ms:1101ms++ 
gc:free=505459k:total=1536000k: 
main:thread exit request: ACE:1528824408481* 
main:detected time out:ACE 
main:restarting collector:ACE 
main:java.lang.OutOfMemoryError 
main:: 
main:Failed to create a thread: retVal -1073741830, errno 11 
........
......
...
mon:thread: ACE has timed out 
mon:thread: ACE has timed out 

 
This scenario has caused the server disk utilization to hit 100%.

Environment

CA Workload Automation iXp - 11.3.5 + version supported platforms

Cause

The resources like disk space and memory are crucial for the tomcat applications.
iXp is not able to create threads normally could be due to hitting the max allocation limits for the process.

Resolution

  • Verify the Basic System Requirements per the iXp documentation
  • Ensure there is enough disk space available in the server
  • Stop the tomcat process
    # $CATALINA_HOME/bin/shutdown.sh



Using CATALINA_BASE:   /opt/CA/iXp/tomcat9
Using CATALINA_HOME:   /opt/CA/iXp/tomcat9
Using CATALINA_TMPDIR: /opt/CA/iXp/tomcat9/temp
Using JRE_HOME:        /opt/CA/iXp/jdk1.8/jre
Using CLASSPATH:       /opt/CA/iXp/tomcat9/bin/bootstrap.jar:/opt/CA/iXp/tomcat9/bin/tomcat-juli.jar



  • Increase the JAVA_OPTS max memory according to the load to meet the memory requirements for iXp.



# export IXP_HOME=/opt/CA/iXp/iXphome
# export JAVA_HOME=/opt/CA/iXp/jdk1.8/jre
# export JAVA_OPTS="-Dixp.home=${IXP_HOME} -Xms1500m -Xmx12000m"
# export CATALINA_HOME=/opt/CA/iXp/tomcat9
# export PATH=$JAVA_HOME/bin:$PATH

 



Important! If the tomcat startup scripts are being used in the environment ensure the changes for the variable JAVA_OPTS are made inside the script.



  • Start the tomcat process​
    # $CATALINA_HOME/bin/startup.sh
    Using CATALINA_BASE:   /opt/CA/iXp/tomcat9
    Using CATALINA_HOME:   /opt/CA/iXp/tomcat9
    Using CATALINA_TMPDIR: /opt/CA/iXp/tomcat9/temp
    Using JRE_HOME:        /opt/CA/iXp/jdk1.8/jre
    Using CLASSPATH:       /opt/CA/iXp/tomcat9/bin/bootstrap.jar:/opt/CA/iXp/tomcat9/bin/tomcat-juli.jar
    Tomcat started.




  • Verify the set limit appears in the process 



# ps -ef|grep ixp
root     16638     1 99 23:31 pts/0    00:00:06 /opt/CA/iXp/jdk1.8/jre/bin/java -Djava.util.logging.config.file=/opt/CA/iXp/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dixp.home=/opt/CA/iXp/iXphome -Xms1500m -Xmx12000m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dignore.endorsed.dirs= -classpath /opt/CA/iXp/tomcat9/bin/bootstrap.jar:/opt/CA/iXp/tomcat9/bin/tomcat-juli.jar -Dcatalina.base=/opt/CA/iXp/tomcat9 -Dcatalina.home=/opt/CA/iXp/tomcat9 -Djava.io.tmpdir=/opt/CA/iXp/tomcat9/temp org.apache.catalina.startup.Bootstrap start
#