AIOPs - Unable to access the DX OI/APM

Products

• DX Application Performance Management
• DX Operational Intelligence
• DX Operational Observability

Issue/Introduction

The following error is reported when trying to access to the UI: HTTP  Error "HTTP ERROR 503 Waiting for a connection with the DB. Try again, shortly"

Environment

• DX APM 2*
• DX OI 2*
• DX O2 2*

Cause

apmservices-apmbacking or some other dx services are not running as expected.
       

Resolution

SUGGESTIONS:

1) Check that apmbacking service is up and running

kubectl get pods -n<namespace> | grep apmbacking

ACTION TO TAKE:

Scale up apmservices-apmbacking service

kubectl scale --replicas=1 deployment apmservices-apmbacking-singleton -n<your-dxi-namespace>

 

2) Check the content of <dx-platform-home>/tools/dx-admin.data is correct

dx-admin.data is created automatically by the product, no action is required by users, but as a general best practice, backup the original <dx-platform-home>/dxi-input-var.yml created after installation to prevent unexpected situations, for example: if this file get corrupted or it is accidentally modified.

ACTION TO TAKE:

Ensure all expected services have been configured with the correct replicates, check if the are lines that contains "0" at the beginning and investigate if the replicas numbers are correct. You will need to collect the <dx-platform-home>/dxi-input-vars.yml file to confirm for example which products have been installed, what is the ElasticSearch type( small or medium),etc, based on this review you can confirm that the # of replicas in dx-admin.data file is correct or not

Example 1:

Below example, illustrating an issue with "apmservices-apmbacking-singleton" , by mistake it has been configured with replicas 0 .

vi <dx-platform-home>/tools/dx-admin.data

ACTION TO TAKE: edit the file, change the 0 replicas to 1, then  stop and restart all services again

 

Example 2: below another example, illustrating the same issue, in this case multiple jarvis and doi services have been configured with replicas 0 incorrectly.

ACTION TO TAKE: update replicas to 1 for elasticsearch-master, jarvis-kafka, jarvis-zookeeper, situations-deployment , then restart all services again. (NOTE: elasticsearch-client and elasticsearch-warm replicas is by default 0, check with your team these #s needs to be manually corrected)

If you are not sure, contact Broadcom Support

 

Additional Information

AIOPs Troubleshooting and Common Issues