UMP USM portlet is stuck loading

Products

• DX Infrastructure Management
• NIMSOFT PROBES

Issue/Introduction

The UMP USM portlet is stuck on a loading progress bar and never loads.

Cause

The udm_manager was failing with the following error that is only seen when the loglevel of this probe is set to 3:

 

Dec 28 13:54:21:991 WARN  [clojure-agent-send-off-pool-1, udm_manager] stderr: Critical failure, cannot continue: Error starting transactor

Dec 28 13:54:21:993 WARN  [clojure-agent-send-off-pool-1, udm_manager] stderr: java.lang.IllegalArgumentException: :db.error/not-enough-memory (datomic.objectCacheMax + datomic.memoryIndexMax) exceeds 75% of JVM RAM

At the default loglevel of 1, the log shows the probe restarting every minute:

 

Dec 28 13:49:51:424 INFO  [main, udm_manager] ****************[ Starting ]****************
Dec 28 13:49:51:424 INFO  [main, udm_manager] 8.40
Dec 28 13:50:44:576 INFO  [main, udm_manager] Additional transactor logs are located under the directory transactor-log.
Dec 28 13:50:44:798 INFO  [main, udm_manager] ****************[ Starting ]****************
Dec 28 13:50:44:798 INFO  [main, udm_manager] 8.40
Dec 28 13:51:38:513 INFO  [main, udm_manager] Additional transactor logs are located under the directory transactor-log.

 

In Infrastructure Manager or Admin Console, you can see the PID of the udm_manager change with each restart.  The probe remains green (active) in Infrastructure Manager and Admin Console even though it fails to to start the transactor due to the "not-enough-memory" error.

The discovery_server log may also show an error similar to the following:

28 Dec 2016 16:16:44,366 [main] WARN  com.nimsoft.udm.utilities.nimbus.UdmConnectionLookup - Waiting to obtain UDM database connection properties: (11) command not found, Received status (11) on response (for sendRcv) for cmd = 'getUdmDatabaseConnection'  Retrying...

This probe will also remain green (active) in Infrastructure Manager and Admin Console.

No alarms are raised by either probe in this situation.

Environment

UIM Server/UMP 8.40 through the 8.47 GA release available at this time

Resolution

Increase the java memory keys for the udm_manager from the probe's Raw Configure GUI.  Make sure there is available memory on the primary hub before increasing the value of the java_mem_max key to accommodate the additional memory allocated to the probe.  It is recommended that you increase the java_mem_max to 4g:

Apply the change.

To implement these changes follow these steps:

1.  Deactivate the following probes in the listed order waiting for the PID and port columns to clear in Infrastructure Manager or Admin Console for the current probe before deactivating the next one:

• wasp
• discovery_server
• trellis
• udm_manager

2.  Activate the same set of probes in reverse order waiting until a value appears in the PID and port columns in Infrastructure Manager or Admin Console for the current probe before activating the next one:

• udm_manager
• trellis
• discovery_server
• wasp