We tried to upgrade our software gateway from 10 to 10.1 server and it is not coming up after the upgrade.
And also below are the error in the controller logs:
2022-05-03T00:28:32.006+0800 INFO 11 com.l7tech.server.processcontroller.ProcessController: default stopping
2022-05-03T00:30:51.894+0800 CONFIG 1 com.l7tech.server.processcontroller: Logging initialized with defaults from 'jar:file:/opt/SecureSpan/Controller/Controller.jar!/com/l7tech/server/processcontroller/resources/logging.properties'
2022-05-03T00:30:51.987+0800 INFO 1 com.l7tech.server.processcontroller.BootstrapConfig: Starting configuration bootstrap Layer7 API Gateway 10.1.00 build 11620, built 20210706173756 by teamcity at ${env.COMPUTERNAME}
2022-05-03T00:30:51.993+0800 INFO 1 com.l7tech.server.processcontroller.BootstrapConfig: Configuration bootstrap complete.
2022-05-03T00:30:52.477+0800 CONFIG 1 com.l7tech.server.processcontroller: Logging initialized with defaults from 'jar:file:/opt/SecureSpan/Controller/Controller.jar!/com/l7tech/server/processcontroller/resources/logging.properties'
2022-05-03T00:30:52.548+0800 INFO 1 com.l7tech.server.processcontroller.ProcessControllerDaemon: Starting Process Controller Layer7 API Gateway 10.1.00 build 11620, built 20210706173756 by teamcity at ${env.COMPUTERNAME}
2022-05-03T00:30:53.160+0800 INFO 1 com.l7tech.server.processcontroller.ConfigServiceImpl: Using java binary: /opt/SecureSpan/JDK/bin/java
2022-05-03T00:30:54.008+0800 INFO 1 com.l7tech.server.processcontroller.ConfigServiceImpl: Remote node management disabled
2022-05-03T00:30:54.009+0800 INFO 1 com.l7tech.server.processcontroller.ConfigServiceImpl: No user trusted certificates for patch verification.
2022-05-03T00:30:54.061+0800 INFO 1 com.l7tech.server.processcontroller.ConfigServiceImpl: Detected node 'default'.
2022-05-03T00:30:54.101+0800 INFO 1 com.l7tech.server.processcontroller.ConfigServiceImpl: No monitoring configuration found in /opt/SecureSpan/Controller/var/currentMonitoringConfig.xml
2022-05-03T00:30:54.203+0800 INFO 1 com.l7tech.server.processcontroller.PCServletContainer: Added default IPv4 SSL connector.
2022-05-03T00:30:55.559+0800 INFO 1 com.l7tech.server.processcontroller.ProcessController: Getting API port from /opt/SecureSpan/Gateway/node/default/var/processControllerPort
2022-05-03T00:30:55.775+0800 INFO 1 com.l7tech.server.processcontroller.p: default starting
2022-05-03T00:31:00.809+0800 INFO 1 com.l7tech.server.processcontroller.ProcessController: Getting API port from /opt/SecureSpan/Gateway/node/default/var/processControllerPort
2022-05-03T00:31:05.825+0800 INFO 1 com.l7tech.server.processcontroller.ProcessController: Getting API port from /opt/SecureSpan/Gateway/node/default/var/processControllerPort
2022-05-03T00:31:10.837+0800 INFO 1 com.l7tech.server.processcontroller.ProcessController: Getting API port from /opt/SecureSpan/Gateway/node/default/var/processControllerPort
2022-05-03T00:31:15.847+0800 WARNING 1 com.l7tech.server.processcontroller.p: default exited with status 77
2022-05-03T00:31:15.909+0800 WARNING 1 com.l7tech.server.processcontroller.ProcessController: default crashed on startup with exit code 77
2022-05-03T00:31:15.910+0800 WARNING 1 com.l7tech.server.processcontroller.ProcessController: default crashed on startup; copying its output:
Release : 10.1
Component :
appdynamics entries added on ssgruntimedefs.sh
# LAYER 7 TECHNOLOGIES
# Defines JAVA_HOME, etc
default_java_opts="-server "
if [ ! -f "/etc/sysconfig/network" -o "$(grep "^NETWORKING_IPV6=yes" /etc/sysconfig/network 2>/dev/null)" ]
then
default_java_opts="$default_java_opts -Djava.net.preferIPv4Stack=false "
else
default_java_opts="$default_java_opts -Djava.net.preferIPv4Stack=true "
fi
default_java_opts="$default_java_opts -Djava.security.policy=${SSG_HOME}/runtime/etc/ssg.policy"
default_java_opts="$default_java_opts -Djava.security.egd=file:/dev/./urandom"
default_java_opts="$default_java_opts -Dfile.encoding=UTF-8 "
default_java_opts="$default_java_opts -Djava.awt.headless=true -XX:CompileThreshold=1500 "
default_java_opts="$default_java_opts -Dcom.l7tech.server.defaultClusterHostname=$(hostname)"
default_java_opts="$default_java_opts -Dcom.sun.management.jmxremote.port=5002"
default_java_opts="$default_java_opts -Dcom.sun.management.jmxremote.rmi.port=5002"
default_java_opts="$default_java_opts -Dcom.sun.management.jmxremote.authenticate=false"
default_java_opts="$default_java_opts -Dcom.sun.management.jmxremote.ssl=false"
# sun.net.inetaddr.ttl and java.util.Arrays.useLegacyMergeSort are system properties that are read during
# JVM start up and need to be specified as command line args to make sure the values are used.
# Do not move sun.net.inetaddr.ttl and java.util.Arrays.useLegacyMergeSort into the system.properties file.
default_java_opts="$default_java_opts -Dsun.net.inetaddr.ttl=30 "
default_java_opts="$default_java_opts -Djava.util.Arrays.useLegacyMergeSort=true"
default_java_opts="$default_java_opts -javaagent:/opt/appdynamics/appagent/javaagent.jar"
default_java_opts="$default_java_opts -Dappdynamics.http.proxyHost=web.prod.proxy.domain.com"
default_java_opts="$default_java_opts -Dappdynamics.http.proxyPort=4200"
default_java_opts="$default_java_opts -Dappdynamics.http.proxyUser=ps123449"
default_java_opts="$default_java_opts -Dappdynamics.http.proxyPasswordFile=/opt/appdynamics/appagent/conf/pswd"
SSG_JAVA_OPTS="$SSG_JAVA_OPTS $default_java_opts";
unset default_java_opts
export SSG_JAVA_OPTS
if ! echo $LD_LIBRARY_PATH | /bin/egrep -s "(^|:)$SSG_HOME/runtime/lib($|:)" >/dev/null ; then
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SSG_HOME/runtime/lib
fi
export LD_LIBRARY_PATH
1.please, ensure you follow this steps when doing the upgrade using patches
Patch installation Order - Proper Patch Install Order
Proper patching for Gateway Upgrade from 10.0 - 10.1 CR01
https://knowledge.broadcom.com/external/article?articleId=234729
2. modified the ssgruntimedefs.sh and leave as follow :
# LAYER 7 TECHNOLOGIES
# Defines JAVA_HOME, etc
default_java_opts="-server "
if [ ! -f "/etc/sysconfig/network" -o "$(grep "^NETWORKING_IPV6=yes" /etc/sysconfig/network 2>/dev/null)" ]
then
default_java_opts="$default_java_opts -Djava.net.preferIPv4Stack=false "
else
default_java_opts="$default_java_opts -Djava.net.preferIPv4Stack=true "
fi
default_java_opts="$default_java_opts -Djava.security.policy=${SSG_HOME}/runtime/etc/ssg.policy"
default_java_opts="$default_java_opts -Djava.security.egd=file:/dev/./urandom"
default_java_opts="$default_java_opts -Dfile.encoding=UTF-8 "
default_java_opts="$default_java_opts -Djava.awt.headless=true -XX:CompileThreshold=1500 "
default_java_opts="$default_java_opts -Dcom.l7tech.server.defaultClusterHostname=$(hostname)"
# sun.net.inetaddr.ttl and java.util.Arrays.useLegacyMergeSort are system properties that are read during
# JVM start up and need to be specified as command line args to make sure the values are used.
# Do not move sun.net.inetaddr.ttl and java.util.Arrays.useLegacyMergeSort into the system.properties file.
default_java_opts="$default_java_opts -Dsun.net.inetaddr.ttl=30 "
default_java_opts="$default_java_opts -Djava.util.Arrays.useLegacyMergeSort=true"
SSG_JAVA_OPTS="$SSG_JAVA_OPTS $default_java_opts";
unset default_java_opts
export SSG_JAVA_OPTS
if ! echo $LD_LIBRARY_PATH | /bin/egrep -s "(^|:)$SSG_HOME/runtime/lib($|:)" >/dev/null ; then
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SSG_HOME/runtime/lib
fi
export LD_LIBRARY_PATH
**If gateway not starting properly after change above , do the following :
1. gateway.sh stop
# find -name ssgruntimedefs.sh
2. once identified the folder , replace your current ssgruntimedefs.sh with the file (ssgruntimedefs.sh from a working v10.1 env) :
please, copy this file to /home/ folder and change the permissions to the file as follow :
chown layer7:layer7 ssgruntimedefs.sh
2. once replaced the ssgruntimedefs.sh on the proper folder, please execute
# su - gateway
# ./opt/SecureSpan/Gateway/runtime/bin/gateway.sh run
3. we will check if gateway is running or writing something to ssg
ps -fea |grep Gateway.jar
3. you can try to start the gateway in console mode, showing all details on the startup process to check if there is some error.
example :
./gateway.sh start -console