High usage of java.exe process at Service Catalog server when integrated to Service Desk Manager

book

Article ID: 6608

calendar_today

Updated On:

Products

CA Service Management - Asset Portfolio Management CA Service Management - Service Desk Manager

Issue/Introduction

Sometimes the Service Catalog's Java process will reach 100% CPU utilization.

During these times the PDM CATALOG SYNC DAEMON (pdm_catalog_sync) can be see back in Service Desk Manager (SDM) server showing as in "Connecting" state.

In ServiceCatalog.log and view.log file you could see below messages appearing:

 

INFO | jvm 1 | YYYY/MM/DD HH:MM:SS | java.lang.OutOfMemoryError: GC overhead limit exceeded 

INFO | jvm 1 | YYYY/MM/DD HH:MM:SS | Dumping heap to ../../logs\java_pid5616.hprof ... 

ERROR | wrapper | YYYY/MM/DD HH:MM:SS | JVM appears hung: Timed out waiting for signal from JVM. 

ERROR | wrapper | YYYY/MM/DD HH:MM:SS | JVM did not exit on request, terminated 

 

STATUS | wrapper | 2017/03/22 15:01:14 | Launching a JVM... 

 

ERROR [localhost-startStop-1] [BrokerService] Temporary Store limit is 51200 mb, whilst the temporary data directory: E:\CA\Catalog\logs\jms-data\brkr-585AE64695A949058FB9DBA1F071EC10\tmp_storage only has 45907 mb of usable space - resetting to maximum available 45907 mb. 

ERROR [localhost-startStop-1] [BrokerService] Failed to start Apache ActiveMQ ([brkr-585AE64695A949058FB9DBA1F071EC10, null], java.io.IOException: Detected missing/corrupt journal files. 4792 messages affected.) 

ERROR [localhost-startStop-1] [ActiveMqBroker] Exception while trying to start JMS broker

Cause

This seems related to Catalog performance and Catalog accepting the SDM connection rather than a problem with the SDM process itself.

This problem could be caused by a build up of files in the ..\Service Catalog\logs\jms-data folder.

This can lead up to memory build up if there are lot of unconsumed JMS events.

Environment

Service Catalog 14.1 / Service Desk Manager 14.1

Resolution

Please take the following steps to remove the files: 

 

1. Stop the CA Service Catalog service

2. Delete Jms-data by making a backup of the folder "..\Service Catalog\logs\jms-data" to a separate folder outside "..\Service Catalog\logs", then delete all the files under "..\Service Catalog\logs\jms-data"

3. Change all from maxBytesLocalHeap="100M" to maxBytesLocalHeap="25M" in ehcache.xml file location \CA\Service Catalog\view\conf of all Node (if all nodes are having the problem otherwise change in problematic node)

4. Start the CA Service Catalog service.

 

Note: Because of the rebuild of the jms message queue, it may take longer for Service Catalog to completely startup. 

 

Once Service Catalog is restarted, monitor the java.exe process for it's CPU and memory usage and check the pdm_catalog_sync daemon process to ensure it is in 'Running' status.

Additional Information

It might also be needed to increase the Maximum Java Heap Size at the Catalog server to something greater than the default value (check at the viewService.conf file for the parameter 'wrapper.java.maxmemory' currently set by default to '1024' or '1Gb', you might need to increase it to something bigger such as '3Gb' or '4Gb').

Note:

It is possible that in order to accommodate the new memory values, there should also be needed to increase the RAM memory at the servers.