The UVMS process uses constantly high load of CPU in large environments

Products

• CA Automic Dollar Universe

Issue/Introduction

Error Message :
In UVMS uvserver.log this kind of error appears a lot of times:
|ERROR| request-worker-pid | com.orsyp.central.server.DispatchHandler | Handle creation error
java.lang.RuntimeException: Invoke flush on output stream failed! See cause:
at com.orsyp.requestprocessor.UvmsRequestProcessor.processRequest(UvmsRequestProcessor.java:53)

The issue causes the UVMS to respond very slowly in general.
UVC connections are impossible sometimes and sometimes they can take up to 3 minutes.

By checking the "UVMS Server Monitoring"  via UVC, a flat line at 100% for the UVMS CPU can be observed. On the OS a constant high CPU load can be seen via the corresponding OS tools (e.g. top for Linux/Unix or TaskManager on Windows) for the UVMS process as well.

Environment

Issue found on UVMS with lot of activity (multiple UVC sessions) and more than 50 v6 nodes.

Cause

Cause type:Defect
Root Cause: Some queries coming from the UVC clients related to the Node List could impact the global performance of the UVMS.

Some additional services like "Consolidation of Job Run Summary" and the DUAS 6 "Synchronization" can also impact performances if performed too frequently in large environments.

Resolution

Upgrade to version 6.8.21 or superior the UVMS where some new node settings on UVMS side have been added to help with the usage of the CPU.
With that version new parameters have been introduced to allow optimizations.

Check the UVMS node settings and verify that the Optional Service "Enable Consolidation of Job Summary" is set to No.
Check as well the Synchronization settings and that the values are not lower than 30 / 90, else set them to these values:

DUAS v6 to UVMS data synchronization cycle (seconds)    360
DUAS v6 to UVMS synchronization cycle (seconds)    90


For UVMS on large environments ( around 100 nodes and multiple users ) we suggest to set those new parameters as follows:

1. Stop the UVMS
2. Backup the file <uvms_folder>/data/values.xml

3. Add the following at the end of the <unijob> section right before the end tag  </unijob> within values.xml:


<unijob>
..
<var id="DB_VALIDATION_BORROW">N</var>
<var id="DB_VALIDATION_IDLE">N</var>
<var id="GET_ALL_NODES_QUERY_TIMEOUT">10000</var>
<var id="GET_ALL_NODES_QUERY_READONLY">Y</var>
<var id="ROLE_CACHE_CLEANUP_INTERVAL">60</var>
</unijob>

4. Restart the UVMS

5. Check that issue (constant high CPU usage) does not reproduce anymore, in case it does, please add or replace the following variables on the values.xml:

<var id="CHECK_NODE_UNREACHABLE">N</var>
<var id="OWLS_UVMS_UVC_NODE_STATUS_TIMER">600</var>
<var id="DB_MAX_POOL_SIZE">500</var>

<var id="AUT_THREAD_MAX">3000</var>

<var id="UXMAXJVMMEM">2048</var>

6. Restart again the UVMS to take into account the new parameters.

In the unlikely case that the CPU utilization still goes to 100% constantly, please let us know.

Additional Information

Workaround :
Restart the UVMS process as soon as it starts to go into the constant high CPU load situation

 

Fix Status: Released
Fix Version(s):
Component: Univiewer Management Server

Fix version: 6.8.21 - Available