DX Platform - Unable to login to DX Platform due to problem with AXA or Postgres services

Products

• DX Application Performance Management
• DX Operational Intelligence
• CA App Experience Analytics

Issue/Introduction

Symptoms:

Cause

Problem with AXA or Postgres services

Environment

DX Operational Intelligence 1.3.x, 20.2
DX Application Performance Management 11.x, 20.2
DX AXA 20.2

Resolution

Option 1: Check Postgres

a) Verify if Postgres pod is running:

kubectl get pods -n<your project> | grep postgres

Example:

kubectl get pods -ndxi | grep postgres
dxi-postgresql-bccdd4475-86z4d                        1/1       Running     0          34d

b) Check for possible errors in available postgres logs:

If DX 20.2:

kubectl log -f <dxi-postgresql-pod> -ndxi

OR

<nfs>/axaservices/pg-data/userdata/pg_log/*

If DOI132 

kubectl log -f <postgresql-pod> -ndxi

OR

<nfs>/pg-data/userdata/pg_log/*

c) Restart or scale down and up the postgres pod as below:

kubectl scale --replicas=0 deployment dxi-postgresql -ndxi

Wait for a minute

kubectl scale --replicas=1 deployment dxi-postgresql -ndxi

d) Check for possible errors in the postgres logs as indicated in step (b)

Option 2: Check readserver and adminui pods

If Postgres is running, the problem could be that ActiveMQ is not able to connect with Readserver, check the logs:

If DX 20.2:

kubectl get pods -n<namespace> | egrep "doi-adminui|dxi-readserver|doireadserver"

kubectl log <doi-adminui-pod> -n<namespace>
kubectl log <dxi-readserver-pod> -n<namespace>
kubectl log <doireadserver-pod> -n<namespace>

OR

-doi-adminui
Mount : <nfs>/doiservices/adminui/logs = /opt/ca/axa/logs
Mount : <nfs>/doiservices/adminui/tomee-logs = /opt/ca/axa/apache-tomee-8.0.0-plus/logs

-dxi-readserver
Mount: <nfs>/axaservices/dxi-readserver/logs = /opt/caemm/logs

-doireadserver 
Mount : <nfs>/doiservices/readserver/logs = /opt/caemm/logs
Mount : <nfs>/doiservices/readserver/tomee-logs =  /opt/caemm/apache-tomee-plus-8.0.0/logs

If DOI 1.3.2: 

kubectl get pods -n<namespace> | egrep "adminui|readserver|doireadserver"

kubectl log <adminui-pod> -n<name-space>
kubectl log <readserver-pod> -n<name-space>
kubectl log <doireadserver-pod> -n<name-space>

OR

-adminui
Mount: <nfs>/adminui-logs /opt/ca/axa/logs/
Mount: <nfs>/adminui-tomee-logs /opt/ca/axa/apache-tomee-1.7.1-plus/logs

-readserver 
Mount: <nfs>/axa-data/readserver-logs = /opt/caemm/logs

-doireadserver
Mount : <nfs>/doi-readserver-logs = /opt/caemm/logs
Mount : <nfs>/doireadserver-tomee-logs = /opt/caemm/apache-tomee-plus-1.7.1/logs

Recommendation: Restart the adminui and readserver pods in the following order:

If DX 20.2:

Scale-down the components in the following order:

kubectl scale --replicas=0 deployment doi-adminui -n<namespace>
kubectl scale --replicas=0 deployment dxi-readserver -n<namespace>
kubectl scale --replicas=0 deployment doireadserver -n<namespace>

Now, you need to scale-up the components in the following order:

kubectl scale --replicas=1 deployment doireadserver -n<namespace>
kubectl scale --replicas=1 deployment dxi-readserver -n<namespace>
kubectl scale --replicas=1 deployment doi-adminui -n<namespace>

If DX 1.3.2:

Scale-down the components in the following order:

kubectl scale --replicas=0 deployment adminui -n<namespace>
kubectl scale --replicas=0 deployment readserver -n<namespace>
kubectl scale --replicas=0 deployment doireadserver -n<namespace>

Now, you need to scale-up the components in the following order:

kubectl scale --replicas=0 deployment doireadserver -n<namespace>
kubectl scale --replicas=0 deployment readserver -n<namespace>
kubectl scale --replicas=0 deployment adminui -n<namespace>

 

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815