spectrumgtw losing connection with udm

Products

DX Unified Infrastructure Management (Nimsoft / UIM) CA Spectrum

Issue/Introduction

We are seeing multiple "lost connection" errors in the spectrumgtw probe logs. Sometimes it causes an exception as well. Please let us know how to fix this from occurring.i.e.

Nov 02 2020 22:57:32,214 [main] ERROR com.nimsoft.nimbus.NimProbe - Lost connection for the "subscribe" connection. Trying to reconnect
Nov 02 2020 22:57:32,249 [main] INFO com.nimsoft.nimbus.NimProbeBase - subscribe to queue alarm failed ((4) not found, Received status (4) on response (for sendRcv) for cmd = 'subscribe' name = 'null'), trying subscribe to subject udm_inventory

Environment

Release : 20.1

Component : UIM - UDM

CA UIM v20.10
Trellis Probe v20.10
CA UIM nas probe v9.30
ems probe v10.23
spectrumgtw probe v8.67hf6
Spectrum UIM Services v8.6.7
CA Spectrum v10.4.1

Resolution

Increase java memory to udm_manager probe and also the timeouts for connecting to the database.

1)Deactivate udm_manager and discovery_server probe

In the " Raw Configure" GUI of the udm_manager, add the following key under the setup section:

"schema_transact_retries" and set the value to 5000.

Also increase:

"schema_connection_timeout_minutes" value to 30.

You may also need to edit this parameter under "setup" > datomic > "memory_index_max".

Out of the box this is set to 256m. You may need to increase this to '512m' or higher.

3. There are settings that are designed to make more memory available to udm_manager

setup/datomic/memory_index_max = 512m
setup/datomic/object_cache_max = 1024m
setup/datomic/heartbeat_interval_msec = 30000
startup/opt/java_mem_max = -Xmx4g

heartbeat_interval_msec needs to be added manually as this key doesn't exists. Click on New Key and add.

4) truncate table datomic_kvs

5) Activate udm_manager probe

6) Activate discovery_server probe

References
=========
udm_inventory queue not processing data

https://knowledge.broadcom.com/external/article?articleId=112962

If none of the above resolves the problem, please navigate to:

https://support.broadcom.com/external/content/release-announcements/CA-Unified-Infrastructure-Management-Hotfix-Index/7233

and download and deploy:

spectrumgtw-8.6.8-HF1.zip

Here are the release notes:

Deployment steps:

1. Download spectrumgtw-8.6.8-HF1.zip and add it to UIM Archive.
2. Deactivate/stop Spectrumgtw probe.
3. Delete/backup existing spectrumgtw probe logs.
4. Deploy the spectrumgtw version 8.6.8-HF1 from Archive.
5. Delete spectrumgtw-8.6.8-SNAPSHOT, if present, and graph-transform-0.0.4 jar files from spectrumgtw/lib path.
6. Activate the spectrumgtw probe.

What's new in spectrumgtw-8.6.8-HF1 :

1. Added new graph-transform-0.0.7 library.
2. Added Post reply timeout configuration QueueReplyTimeoutInSeconds with default value 1200 seconds.
3. Added configuration to enable and disable VMotion Sync with default value true means VMotionSync enabled.
4. Modified queue subscription mechanism.
5. QueueMonitoring:
a) Added Queue monitor thread interval configuration QueueMonitorIntervalInSeconds with default value 10 seconds.
b) Monitoring queue subscription at regular interval of QueueMonitorIntervalInSeconds and re-subscribing to queue if the queue is disconnected.

Note that spectrumgtw-8.6.8-HF1 has only been tested with Spectrum v10.4.2. In this case it still worked and the connection issues were resolved and alarms were still syncing to Spectrum.