AIOps - After creating a new tenant jarvis-verifier pod keeps crashing

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

Symptoms:

- The problem started after attempting to create a New Tenant with max capacity, no error was reporting during Tenant creation.

- jarvis-verifier pod keeps restarting and from the logs, we can see the process is crashing 

kubectl logs <jarvis-verifier-pod> -n <your-namespace> 

Here is an example:...

2021-05-31 08:00:58 INFO  [pool-4-thread-1] ESClusterManager:169 - Defaulting to primary ES for product: ao and tenant: 
2021-05-31 08:00:58 INFO  [pool-4-thread-1] ESClusterManager:482 - Successfully updated cache for productId : ao and tenantId: 
startVerifier.sh: line 185:    64 Killed                  java -Xms$min_mem_limit -Xmx$max_mem_limit -XX:SurvivorRatio=16 -Dlog4j.configurationFile="file:/opt/ca/verifier/config/verifier.log4j.properties" -jar verifier.jar

 

Environment

DX Operational Intelligence 2x
DX Application Performance Management 2x
DX AXA 2x

Cause

Lack of allocated memory affecting the jarvis-verifier process.

Resolution

Increase memory of jarvis-verifier deployment as below:

1) kubectl edit deployment jarvis-verifier -n<your-namespace>

2) Scroll down and change default memory values from 1G to 4G

 

3. Save the file -- this change will restart a new jarvis-verifier pod

 

4. jarvis-verified should be running as expected

 

 

NOTE: If the problem persists, collect Jarvis logs  (elastic, zookeeper, kafka, apis logs,  see below KB) and contact Broadcom Support for assistance:

 

DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119

 

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815