Landscape frequently failing over to secondary despite primary up and healthy and another OneClick server having no failed heartbeats.

Products

• Network Observability
• Spectrum

Issue/Introduction

We have one of our landscapes that has frequently been failing over to it's secondary every few hours the last 2 days and causing big floods of alarms due to devices that aren't correctly configured to respond to the secondary.

 

Then, generally within about a minute the primary takes back over.

 

We don't see any issues on the primary server, and when we check the OneClick in the same DataCenter as the primary it hasn't had a heartbeat failure since early November.

Environment

 Spectrum :: All Supported Versions

Cause

We see this entry in the stdout.log showing a lost heartbeat

 

Jan 02, 2026 14:42:52.832 (CORBAMonitorPool-76638: PollTask for <HOSTNAME>\SnmpServ:<USER>@<HOSTNAME>: ) (CORBAOBJMON) - <HOSTNAME>\SnmpServ: Out checkConnection: false

Jan 02, 2026 14:42:52.832 (CORBAMonitorPool-76638: PollTask for <HOSTNAME>\SnmpServ:<USER>@<HOSTNAME>: ) (CORBAOBJMON) - <HOSTNAME>\SnmpServ:  IN switchConnectionUsingPolledObjects

Jan 02, 2026 14:42:52.832 (CORBAMonitorPool-76638: PollTask for <HOSTNAME>\SnmpServ:<USER>@<HOSTNAME>: ) (CORBAOBJMON) - <HOSTNAME>\SnmpServ: In checkConnection, timeout: 60

Jan 02, 2026 14:42:52.832 (CORBAMonitorPool-76638: PollTask for <HOSTNAME>\SnmpServ:<USER>@<HOSTNAME>: ) (CORBAOBJMON) - <HOSTNAME>\SnmpServ: Out checkConnection: false

 

Resolution

Since only 1 heartbeat is lost and the next one is successful, and since this only affects 1 oneclick in the environment, this seems to be a network issue.

Additional Information

In 22.2.5, broadcom added additional tracing for network related issues that is triggered when Corba debug is enabled

To enable the debug:

Please launch the Spectrum webpage.

Go to the administration tab, then in the grey bar at the top select debug, then on the left, select web server debug page (runtime).

Scroll down to

Corba object monitor

Corba helper

At the bottom, hit apply.

 

Debug is written to the tomcat log

(stdout.log on windows, catalina.out on Linux)

 

   Alternate method of enabling the debug:

  How to enable Corba debug on Tomcat startup

Once the network lag or disconnect is seen by Spectrum services, look for this in the tomcat log:

Triggering network diagnostics for connectivity to <HOSTNAME> :

And review the information that follows in the log.