DX Platform - Unable to login due to PODs connectivity issues

Products

• DX Operational Intelligence
• CA App Experience Analytics
• DX Application Performance Management

Issue/Introduction

Symptoms:

 

From developer tools we can see: 

{"error":{"code":0,"message":"GENERIC_SERVICE_ERROR","traceId":"19e3aad71bcb70d7"}}

 

 

From DX 20.2 > apmservices-manager log: 

2021-04-20 13:40:31.163 ERROR [manager,a5fe2d0170c1715c,f509b08f79cf2a0b,false] 1 --- [nio-8008-exec-4] c.c.a.c.rest.ServiceExceptionHandler     : /dxiess/login, 500,0,a5fe2d0170c1715c: GENERIC_SERVICE_ERROR, null

com.ca.apm.common.api.ServicesException: 500,0,a5fe2d0170c1715c: GENERIC_SERVICE_ERROR, null
 at com.ca.apm.common.api.ServicesException.getInstance(ServicesException.java:202) ~[apmservices.api-20.7.0.3.jar:na]

 

 

Cause

Network issues affecting pods connectivity

Environment

DX Operational Intelligence 1.3.x, 20.2, 
DX Application Performance Management 11.x, 20.2
DX AXA 20.2

Resolution

If DX 20.2+

1) Scale down the following deployments:

kubectl scale --replicas=0 deployment doi-automic-integration -n<namespace>
kubectl scale --replicas=0 deployment doi-adminui -n<namespace>
kubectl scale --replicas=0 deployment doireadserver -n<namespace>
kubectl scale --replicas=0 deployment apmservices-manager-001 -n<namespace>
kubectl scale --replicas=0 deployment dxi-adminui -n<namespace>
kubectl scale --replicas=0 deployment dxi-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-scheduler -n<namespace>
kubectl scale --replicas=0 deployment axaservices-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-amq -n<namespace>
kubectl scale --replicas=0 deployment dxi-postgresql -n<namespace>

2) Verify that all pods have been terminated

kubectl get pods -ndxi | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-scheduler|axaservices-readserver|axaservices-amq|dxi-postgresql"

 

3) Scale up deployments in the following order 1 by 1.

IMPORTANT: Make sure all pods are start correctly by checking the pods logs using : kubectl logs -f <pod-name> -n <namespace>

as below:

kubectl scale --replicas=1 deployment dxi-postgresql -n<namespace>

Wait for below lines in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment axaservices-amq -n<namespace>

Wait for below lines in the pod log before starting the next pod:

 

kubectl scale --replicas=1 deployment axaservices-readserver -n<namespace>

The start of this process can take several minutes

kubectl scale --replicas=1 deployment axaservices-scheduler -n<namespace>

Wait for below lines in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment dxi-readserver -n<namespace>

The start of this process can take several minutes

Wait for below lines in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment dxi-adminui -n<namespace>

Wait for below lines in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment apmservices-manager-001 -n<namespace>

Wait for below lines before starting the next pod:

kubectl scale --replicas=1 deployment doireadserver -n<namespace>

The start of this process can take several minutes

Wait for below lines in the pod log before starting the next pod:

 

kubectl scale --replicas=1 deployment doi-adminui -n<namespace>

Confirm you can see below lines in the pod log:

kubectl scale --replicas=1 deployment doi-automic-integration -n<namespace>

Confirm you can see below lines in the pod log:

3) Finally verify that all pods are up and running:

kubectl get pods -ndxi | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-scheduler|axaservices-readserver|axaservices-amq|dxi-postgresql"

For example:

4) Login to DX UI

 

If DX 1.3.2:

1) Scale down the following deployments:

kubectl scale --replicas=0 dc adminui -n<namespace>
kubectl scale --replicas=0 dc readserver -n<namespace>
kubectl scale --replicas=0 dc doireadserver -n<namespace>
kubectl scale --replicas=0 dc amq -n<namespace>
kubectl scale --replicas=0 dc postgres -n<namespace>

2) Scale up deployments in the following order 1 by 1
NOTE:  ensure all pods are starting correctly by checking the pods logs using : kubectl logs -f <pod-name> -n <namespace>

kubectl scale --replicas=1 dc doi-postgres -n<namespace>
kubectl scale --replicas=1 dc amq -n<namespace>
kubectl scale --replicas=1 dc doireadserver -n<namespace>
kubectl scale --replicas=1 dc readserver -n<namespace>
kubectl scale --replicas=1 dc adminui -n<namespace>

3) Verify that all pods are up and running

4) Login to DOI UI

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815