DevTest Broker Service not starting properly and Agents do not show up on the Broker Console in DevTest Portal.

Products

• CA Cloud Test Mobile
• CA Application Test

Issue/Introduction

We are on Devtest10.6 version and we are facing issue while starting broker .Devtest10.6 is running since yr now and broker was working fine but all of a sudden i am facing below issue. I tried restarting all components still no luck. Please find error from logs.

2020-08-27 06:52:32,942Z (14:52) [ActiveMQ Session Task] INFO  System.out                     - [DevTest AGENT:BC][INFO][23384][69][ActiveMQ Session Task][Aug 27, 2020 2:52:32 PM] Agent online: [ID: 1392575223][NAME: Broker:[email protected]][IP: xxx.xxx.xxx.xxx][MC: com.install4j.runtime.launcher.UnixLauncher][PID: 23384]
2020-08-27 06:52:34,665Z (14:52) [main] INFO  com.itko.lisa.coordinator.EmbeddedWebServer - Starting embedded webserver...
2020-08-27 06:52:34,665Z (14:52) [main] INFO  System.out                     - DevTest Web Consoles will take a few additional moments to start up.
2020-08-27 06:52:34,722Z (14:52) [Thread-29] INFO  com.itko.lisa.pf.PfPathJMSMessageListener - Solr: Create index
2020-08-27 06:52:34,723Z (14:52) [Thread-32] INFO  com.itko.lisa.coordinator.EmbeddedWebServer - The max threads in Server Connector thread pool is 100
2020-08-27 06:52:34,743Z (14:52) [Thread-29] INFO  com.ca.ad.common.solr.LisaEmbeddedSolrServer - Performing Delta-Import
2020-08-27 06:52:34,746Z (14:52) [Thread-32] INFO  com.itko.lisa.coordinator.EmbeddedWebServer - loading /opt/disk1/Lisa/env7/DevTest10.6/webserver/broker/solrservice
2020-08-27 06:52:35,842Z (14:52) [Thread-29] INFO  com.ca.ad.common.solr.LisaEmbeddedSolrServer - import completed
2020-08-27 06:52:49,200Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     - Exception in thread "Pathfinder transaction cleaner" java.lang.OutOfMemoryError: Java heap space
2020-08-27 06:52:49,202Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at java.util.Arrays.copyOf(Arrays.java:3332)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at java.lang.StringBuilder.append(StringBuilder.java:136)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionPersister.deleteBatch(TransactionPersister.java:1463)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionPersister.clearTransactionsOlderThanV2(TransactionPersister.java:1414)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionPersister.clearTransactionsOlderThan(TransactionPersister.java:1319)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionCleaner.cleanUpBasedOnAge(TransactionCleaner.java:93)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionCleaner.executeCleanup(TransactionCleaner.java:46)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionCleaner.access$000(TransactionCleaner.java:17)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err                     -  at com.itko.lisa.remote.persist.TransactionCleaner$1.run(TransactionCleaner.java:30)
2020-08-27 06:52:49,245Z (14:52) [Thread-32] INFO  /solrservice                   - 2 Spring WebApplicationInitializers detected on classpath
2020-08-27 06:52:50,195Z (14:52) [Thread-32] INFO  com.ca.pf.invoke.ApplicationLoader - Starting ApplicationLoader on machine.hc.cloud.hk.abcd with PID 23384 (/opt/disk1/Lisa/env7/DevTest10.6/webserver/broker/solrservice/WEB-INF/classes started by zlisaa1f in /opt/disk1/Lisa/env7/DevTest10.6)
2020-08-27 06:52:50,195Z (14:52) [Thread-32] INFO  com.ca.pf.invoke.ApplicationLoader - The following profiles are active: production
2020-08-27 06:52:54,424Z (14:52) [Thread-32] INFO  /solrservice                   - Initializing Spring embedded WebApplicationContext
2020-08-27 06:52:58,320Z (14:52) [Thread-32] INFO  com.ca.pf.invoke.ApplicationLoader - Started ApplicationLoader in 8.849 seconds (JVM running for 41.84)
2020-08-27 06:52:58,352Z (14:52) [Thread-32] INFO  /solrservice                   - Initializing Spring FrameworkServlet 'grailsDispatcherServlet'

Environment

Release : 10.6

Component : CA Service Virtualization

Cause

This has been fixed as a product defect.

Resolution

Open a support case and refer our internal ticket number DE485659

You will have a patch uploaded to the support case.

Steps to apply:

1. Stop Broker service if it is running

2. Place the patch file "patch_DE485659_10.6.0_GA.jar" at LISA_HOME/lib/patches folder

3. Start Broker service

How to test if the patch has addressed the reported issue:

1. Open Portal and go to Settings -> Agents. Check if you are able to see the agents as it used to appear before the issue was reported

2. Open Pfbroker.log file and make sure the exception message below doesn't appear with the patch in place:

2020-08-27 06:52:49,200Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - Exception in thread "Pathfinder transaction cleaner" java.lang.OutOfMemoryError: Java heap space
2020-08-27 06:52:49,202Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at java.util.Arrays.copyOf(Arrays.java:3332)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at java.lang.StringBuilder.append(StringBuilder.java:136)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionPersister.deleteBatch(TransactionPersister.java:1463)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionPersister.clearTransactionsOlderThanV2(TransactionPersister.java:1414)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionPersister.clearTransactionsOlderThan(TransactionPersister.java:1319)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionCleaner.cleanUpBasedOnAge(TransactionCleaner.java:93)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionCleaner.executeCleanup(TransactionCleaner.java:46)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionCleaner.access$000(TransactionCleaner.java:17)
2020-08-27 06:52:49,203Z (14:52) [Pathfinder transaction cleaner] ERROR System.err - at com.itko.lisa.remote.persist.TransactionCleaner$1.run(TransactionCleaner.java:30)