Multiple EP agent failures, some will not restart

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

After installing EP agents on Windows nodes, Some EPagents have stopped other WILL NOT restart.

EPA logs shows 
2018/07/27 13:31:42 | STATUS | Java Command Line: 
2018/07/27 13:31:42 | STATUS | Command: "E:\Java\jdk\bin\java" -Dcom.wily.introscope.epagent.properties="../IntroscopeEPAgent.properties" -Dfile.encoding=Cp1252 -Duser.language=en -Djava.library.path="../lib" -classpath "../lib/wrapper.jar" -Dwrapper.key="..." -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=8636 -Dwrapper.version="3.5.25-st" -Dwrapper.native_library="wrapper" -Dwrapper.arch="x86" -Dwrapper.service="TRUE" -Dwrapper.cpu.timeout="10" -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.tanukisoftware.wrapper.WrapperJarApp ../lib/EPAgent.jar 
2018/07/27 13:31:42 | ERROR | JVM exited while loading the application. 
 

Cause

Change Java wrapper.JAVA_Home env variable

Environment

APM 10.x

Resolution

To solve the problem:, force the APM EPAgent java wrapper to use 
set.JAVA_HOME=E:\Java\jre8 

Instead of the env JAVA_HOME = 

What the Logs revealed: 
Looking at 
 https://stackoverflow.com/questions/6378579/jvm-error-for-java-service-wrapper 
Your application is throwing some exception while starting up. You need to find a log of what's being thrown, or else set an catch-all exception handler for the program to print a log. 

Key properties in EPAService.conf 

# Enables Debug output from the Wrapper. 
# wrapper.debug=TRUE -- uncomment !!
# Log Level for log file output. (See https://wrapper.tanukisoftware.com/doc/english/prop-logfile-loglevel.html 
for log levels) 
wrapper.logfile.loglevel=STATUS -- Set to DEBUG!!!! 


The DEBUG log revealed
Launching a JVM...
2018/07/27 17:54:10 | DEBUG  | Java Command Line (Query Java Version):
2018/07/27 17:54:10 | DEBUG  |   Command: "E:\Java\jdk\bin\java" -version
2018/07/27 17:54:10 | STATUS | Java Command Line:
2018/07/27 17:54:10 | STATUS |   Command: "E:\Java\jdk\bin\java" -Dcom.wily.introscope.epagent.properties="../IntroscopeEPAgent.properties" -Dfile.encoding=Cp1252 -Duser.language=en -Djava.library.path="../lib" -classpath "../lib/wrapper.jar" -Dwrapper.key="-..." -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.debug="TRUE" -Dwrapper.pid=9580 -Dwrapper.version="3.5.25-st" -Dwrapper.native_library="wrapper" -Dwrapper.arch="x86" -Dwrapper.service="TRUE" -Dwrapper.cpu.timeout="10" -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.tanukisoftware.wrapper.WrapperJarApp ../lib/EPAgent.jar
2018/07/27 17:54:10 | DEBUG  | JVM started (PID=9348)
2018/07/27 17:54:10 | INFO   | Error occurred during initialization of VM
2018/07/27 17:54:10 | INFO   | java/lang/NoClassDefFoundError: java/lang/Object
2018/07/27 17:54:10 | INFO   | Error occurred during initialization of VM
2018/07/27 17:54:10 | INFO   | java/lang/NoClassDefFoundError: java/lang/Object
2018/07/27 17:54:11 | DEBUG  | JVM process exited with a code of 1, setting the wrapper exit code to 1.
2018/07/27 17:54:11 | ERROR  | JVM exited while loading the application.