DX Platform - After creating a new tenant jarvis-verifier keeps crashing

book

Article ID: 216379

calendar_today

Updated On:

Products

DX Operational Intelligence DX Application Performance Management CA App Experience Analytics

Issue/Introduction

Symptoms:

- The problem started after attempting to create a New Tenant with max capacity, no error was reporting during Tenant creation.

- jarvis-verifier pod keeps restarting and from the logs, we can see the process is crashing 

kubectl logs <jarvis-verifier-pod> -n <your-namespace> 

Here is an example:...

2021-05-31 08:00:58 INFO  [pool-4-thread-1] ESClusterManager:169 - Defaulting to primary ES for product: ao and tenant: 03eb6f46-09cc-42a7-a90c-f2d2eba5e766
2021-05-31 08:00:58 INFO  [pool-4-thread-1] ESClusterManager:482 - Successfully updated cache for productId : ao and tenantId: 03eb6f46-09cc-42a7-a90c-f2d2eba5e766
startVerifier.sh: line 185:    64 Killed                  java -Xms$min_mem_limit -Xmx$max_mem_limit -XX:SurvivorRatio=16 -Dlog4j.configurationFile="file:/opt/ca/verifier/config/verifier.log4j.properties" -jar verifier.jar

 

Cause

Lack of allocated memory affecting the jarvis-verifier process.

Environment

DX Operational Intelligence 2x
DX Application Performance Management 2x
DX AXA 2x

Resolution

Increase memory of jarvis-verifier deployment as below:

1) kubectl edit deployment jarvis-verifier -n<your-namespace>

2) Scroll down and change default memory values from 1G to 4G

 
3. Save the file -- this change will restart a new jarvis-verifier pod
 
4. jarvis-verified should be running as expected
 
 
NOTE: If the problem persists, collect Jarvis logs  (elastic, zookeeper, kafka, apis logs,  see below KB) and contact Broadcom Support for assistance:
 
DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119
 

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Attachments