AIOps - Unable to login due to PODs connectivity issues

Products

• DX Operational Intelligence
• CA App Experience Analytics
• DX Application Performance Management

Issue/Introduction

Symptoms:

 

From developer tools we can see: 

{"error":{"code":0,"message":"GENERIC_SERVICE_ERROR","traceId":"19e3aad71bcb70d7"}}

 

 

From DX 20.2 > apmservices-manager log: 

2021-04-20 13:40:31.163 ERROR [manager,a5fe2d0170c1715c,f509b08f79cf2a0b,false] 1 --- [nio-8008-exec-4] c.c.a.c.rest.ServiceExceptionHandler     : /dxiess/login, 500,0,a5fe2d0170c1715c: GENERIC_SERVICE_ERROR, null

com.ca.apm.common.api.ServicesException: 500,0,a5fe2d0170c1715c: GENERIC_SERVICE_ERROR, null
 at com.ca.apm.common.api.ServicesException.getInstance(ServicesException.java:202) ~[apmservices.api-20.7.0.3.jar:na]

 

 

Cause

Possible reasons:

1) Network issues affecting pods connectivity (as per requirements you must ensure Network speed  of 10 Gbps between all nodes)

2) Poor NFS throughput (IOPs and speed), for validation steps refer to https://knowledge.broadcom.com/external/article/212399)

 

 

Environment

DX Operational Intelligence 20.2, 21.3
DX Application Performance Management 20.2, 21.3
DX AXA 20.2, 21.3

Resolution

Table of Contents

• Option 1: Restart all services
• Option 2: Restart only the pods required for the login process
  • For 21.3.x 
  • For 20.2.x 

Option 1: Restart all services

cd <DXPlatform-Installer-HOME>/tools

./dx-admin.sh stop 

Check all pods have been terminated using: kubectl get pods -n<namespace> 

./dx-admin.sh start

Check all pods are up and running using: kubectl get pods -n<namespace> 

Option 2: Restart only the pods required for the login process

 

For 21.3.x 

1) Scale down the following deployments:

kubectl scale --replicas=0 deployment doi-automic-integration -n<namespace>
kubectl scale --replicas=0 deployment doi-adminui -n<namespace>
kubectl scale --replicas=0 deployment doireadserver -n<namespace>
kubectl scale --replicas=0 deployment apmservices-manager-001 -n<namespace>
kubectl scale --replicas=0 deployment dxi-adminui -n<namespace>
kubectl scale --replicas=0 deployment dxi-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-amq -n<namespace>
kubectl scale --replicas=0 deployment dxi-postgresql -n<namespace>

2) Verify that all pods have been terminated

kubectl get pods -n<namespace> | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-readserver|axaservices-amq|dxi-postgresql"

3) Scale up deployments in the following order 1 by 1.

IMPORTANT: Make sure all pods are start correctly by checking the pods logs using : kubectl logs -f <pod-name> -n <namespace>

as below:

kubectl scale --replicas=1 deployment dxi-postgresql -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment axaservices-amq -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

 

kubectl scale --replicas=1 deployment axaservices-readserver -n<namespace>

This process can take several minutes to startup, wait for below lines to appear in the pod log before starting the next pod:

NOTE:  you can ignore the below ERROR messages:

[EMMCacheRefresher,mdo-serverCacheRefreshCheck] ERROR [] - TID[14015724] 5271596: JMS Request processing timeout, please wait for background processing
com.ca.emm.corejsvr.ExceptionWithNC: 5271596: JMS Request processing timeout, please wait for background processing

..

2021-11-30 14:03:09,521 [EMMCacheRefresher,mdo-serverCacheRefreshCheck] ERROR [] - TID[14015724] 3011599: Internal Error: Unable to complete this cache refresh cycle: 5271596: JMS Request processing timeout, please wait for background processing

kubectl scale --replicas=1 deployment dxi-readserver -n<namespace>

This process can take several minutes to startup, wait for below lines to appear in the pod log before starting the next pod.

In addition search for:  Successfully connected to tcp://axaservices-amq:61616

kubectl scale --replicas=1 deployment dxi-adminui -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment apmservices-manager-001 -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment doireadserver -n<namespace>

This process can take several minutes to startup, wait for below lines to appear in the pod log before starting the next pod. 

         Search for the "Server startup" log entry

 

kubectl scale --replicas=1 deployment doi-adminui -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment doi-automic-integration -n<namespace>

Wait for below lines to appear in the pod log:

3) Finally verify that all pods are up and running:

kubectl get pods -n<namespace> | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-readserver|axaservices-amq|dxi-postgresql"

For example:

 

For 20.2.x 

1) Scale down the following deployments:

kubectl scale --replicas=0 deployment doi-automic-integration -n<namespace>
kubectl scale --replicas=0 deployment doi-adminui -n<namespace>
kubectl scale --replicas=0 deployment doireadserver -n<namespace>
kubectl scale --replicas=0 deployment apmservices-manager-001 -n<namespace>
kubectl scale --replicas=0 deployment dxi-adminui -n<namespace>
kubectl scale --replicas=0 deployment dxi-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-scheduler -n<namespace>
kubectl scale --replicas=0 deployment axaservices-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-amq -n<namespace>
kubectl scale --replicas=0 deployment dxi-postgresql -n<namespace>

2) Verify that all pods have been terminated

kubectl get pods -n<namespace> | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-scheduler|axaservices-readserver|axaservices-amq|dxi-postgresql"

3) Scale up deployments in the following order 1 by 1.

IMPORTANT: Make sure all pods are start correctly by checking the pods logs using : kubectl logs -f <pod-name> -n <namespace>

as below:

kubectl scale --replicas=1 deployment dxi-postgresql -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment axaservices-amq -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

 

kubectl scale --replicas=1 deployment axaservices-readserver -n<namespace>

This process can take several minutes to startup, wait for below lines to appear in the pod log before starting the next pod:

NOTE:  you can ignore the below ERROR messages:

[EMMCacheRefresher,mdo-serverCacheRefreshCheck] ERROR [] - TID[14015724] 5271596: JMS Request processing timeout, please wait for background processing
com.ca.emm.corejsvr.ExceptionWithNC: 5271596: JMS Request processing timeout, please wait for background processing

..

2021-11-30 14:03:09,521 [EMMCacheRefresher,mdo-serverCacheRefreshCheck] ERROR [] - TID[14015724] 3011599: Internal Error: Unable to complete this cache refresh cycle: 5271596: JMS Request processing timeout, please wait for background processing

 

kubectl scale --replicas=1 deployment axaservices-scheduler -n<namespace>     

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment dxi-readserver -n<namespace>

This process can take several minutes to startup, wait for below lines to appear in the pod log before starting the next pod

In addition search for:  Successfully connected to tcp://axaservices-amq:61616

kubectl scale --replicas=1 deployment dxi-adminui -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment apmservices-manager-001 -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment doireadserver -n<namespace>

This process can take several minutes to startup, wait for below lines to appear in the pod log before starting the next pod

Search for the "Server startup" log entry

 

kubectl scale --replicas=1 deployment doi-adminui -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

kubectl scale --replicas=1 deployment doi-automic-integration -n<namespace>

Wait for below lines to appear in the pod log:

3) Finally verify that all pods are up and running:

kubectl get pods -n<namespace> | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-scheduler|axaservices-readserver|axaservices-amq|dxi-postgresql"

For example:

4) Login to DX UI

 

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815