Gateway upgrade WONT_START

Products

• CA API Gateway

Issue/Introduction

Multi node cluster upgrade from Gateway 11.0 to 11.1.1.

Gateway status WONT_START

The error log /opt/SecureSpan/Gateway/node/default/var/logs/ssg_0_0.log error message:

2025-01-16T21:52:38.290+0000 SEVERE  1  com.hazelcast.internal.cluster.impl.TcpIpJoiner: [<ipaddress_gateway>]:<hz_port_gateway> [gateway] [5.2.1] Hazelcast instance is not active!
2025-01-16T21:52:38.291+0000 SEVERE  1  com.hazelcast.instance.impl.Node: [<ipaddress_gateway>]:<hz_port_gateway> [gateway] [5.2.1] Could not join cluster. Shutting down now!
2025-01-16T21:52:38.310+0000 WARNING 1  com.l7tech.external.assertions.hazelcastembeddedprovider.server.EmbeddedHazelcastLoader: Could not create Hazelcast instance gateway-hazelcast : Node failed to start!
2025-01-16T21:52:38.310+0000 WARNING 1  com.l7tech.external.assertions.hazelcastembeddedprovider.server.EmbeddedHazelcastLoader: Failed to initialize Hazelcast cluster. java.lang.IllegalStateException: Node failed to start!
2025-01-16T21:52:38.311+0000 INFO    1  com.l7tech.server.extension.registry.lifecycle.LifecycleExtensionRegistry: Unregistered extension with key com.l7tech.external.assertions.hazelcastembeddedprovider.server.EmbeddedHazelcastLoader
2025-01-16T21:52:38.311+0000 INFO    1  com.l7tech.external.assertions.hazelcastembeddedprovider.server.EmbeddedHazelcastLoader: Hazelcast instance is already stopped.
2025-01-16T21:52:38.311+0000 INFO    1  com.l7tech.external.assertions.hazelcastembeddedprovider.server.EmbeddedHazelcastLoader: Unloaded EmbeddedHazelcastSharedStateProviderAssertion
2025-01-16T21:52:38.311+0000 WARNING 1  com.l7tech.server.boot.GatewayMain: Error starting server : Lifecycle error: Could not initialize Hazelcast cluster
2025-01-16T21:52:38.311+0000 WARNING 1  STDERR: com.l7tech.server.LifecycleException: Lifecycle error: Could not initialize Hazelcast cluster
2025-01-16T21:52:38.311+0000 WARNING 1  STDERR:         at com.l7tech.server.LifecycleExtensionComponent.start(Unknown Source)
2025-01-16T21:52:38.311+0000 WARNING 1  STDERR:         at com.l7tech.server.BootProcess.start(Unknown Source)
2025-01-16T21:52:38.311+0000 WARNING 1  STDERR:         at com.l7tech.server.boot.GatewayBoot.f(Unknown Source)
2025-01-16T21:52:38.311+0000 WARNING 1  STDERR:         at com.l7tech.server.boot.GatewayBoot.start(Unknown Source)
2025-01-16T21:52:38.311+0000 WARNING 1  STDERR:         at com.l7tech.server.boot.GatewayBoot.runUntilShutdown(Unknown Source)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.l7tech.server.boot.GatewayMain.main(Unknown Source)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR: Caused by: com.ca.apim.gateway.extension.lifecycle.LifecycleException: Could not initialize Hazelcast cluster
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.l7tech.external.assertions.hazelcastembeddedprovider.server.a.start(Unknown Source)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         ... 6 more
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR: Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: Node failed to start!
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.l7tech.external.assertions.hazelcastembeddedprovider.server.EmbeddedHazelcastLoader.a(Unknown Source)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         ... 7 more
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR: Caused by: java.lang.IllegalStateException: Node failed to start!
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:196)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:134)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61)
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:         ... 8 more
2025-01-16T21:52:38.312+0000 WARNING 1  STDERR:

**** Unable to start the server: Error starting server : Lifecycle error: Could not initialize Hazelcast cluster

 

Environment

Gateway 11.X.X

Cause

The cluster_info table does not hold version info.   It is a dynamic table that holds info for gateways in the cluster,  when a gateway starts the first thing it does is add it to the table then connects to the Hazelcast cluster.  This is where the version problem occurs.

Resolution

Once all cluster GW are updated 

Stop all nodes on the Layer7 API Gateway:
Access the Gateway main menu.
Select option 2 (Display Layer7 API Gateway configuration menu).
Select option 7 (Manage Layer7 API Gateway status)
Press Enter and then select the option to stop the Gateway.
Repeat these steps on each Gateway node.

Clear cluster_info table 

# mysql

To view: mysql> select * from ssg.cluster_info \G;

Truncate: mysql> truncate table ssg.cluster_info;

 

Then from the SSGCONFIG menu  start each one from option 2 / 7