ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

AIOps - Unable to login due to PODs connectivity issues

book

Article ID: 185182

calendar_today

Updated On:

Products

DX Operational Intelligence CA App Experience Analytics DX Application Performance Management

Issue/Introduction

Symptoms:


 
From developer tools we can see: 
{"error":{"code":0,"message":"GENERIC_SERVICE_ERROR","traceId":"19e3aad71bcb70d7"}}
 
 
From DX 20.2 > apmservices-manager log: 

2021-04-20 13:40:31.163 ERROR [manager,a5fe2d0170c1715c,f509b08f79cf2a0b,false] 1 --- [nio-8008-exec-4] c.c.a.c.rest.ServiceExceptionHandler     : /dxiess/login, 500,0,a5fe2d0170c1715c: GENERIC_SERVICE_ERROR, null

com.ca.apm.common.api.ServicesException: 500,0,a5fe2d0170c1715c: GENERIC_SERVICE_ERROR, null
 at com.ca.apm.common.api.ServicesException.getInstance(ServicesException.java:202) ~[apmservices.api-20.7.0.3.jar:na]
 
 

Cause

Possible reasons:

1) Network issues affecting pods connectivity (as per requirements you must ensure Network speed  of 10 Gbps between all nodes)

2) Poor NFS throughput (IOPs and speed), for validation steps refer to https://knowledge.broadcom.com/external/article/212399)

 

 

Environment

DX Operational Intelligence 20.2, 
DX Application Performance Management 11.x, 20.2
DX AXA 20.2

Resolution

For DX 20.2.x and higher

Option 1: Restart all services

<DXPlatform-Installer-HOME>/tools/dx-admin.sh stop 

Check all pods have been terminated: kubectl get pods -n<namespace> 

<DXPlatform-Installer-HOME>/tools/dx-admin.sh start

Check all pods are up and running: kubectl get pods -n<namespace> 

Option 2: Try to restart only the services required for the login process:

1) Scale down the following deployments:

kubectl scale --replicas=0 deployment doi-automic-integration -n<namespace>
kubectl scale --replicas=0 deployment doi-adminui -n<namespace>
kubectl scale --replicas=0 deployment doireadserver -n<namespace>
kubectl scale --replicas=0 deployment apmservices-manager-001 -n<namespace>
kubectl scale --replicas=0 deployment dxi-adminui -n<namespace>
kubectl scale --replicas=0 deployment dxi-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-scheduler -n<namespace>
kubectl scale --replicas=0 deployment axaservices-readserver -n<namespace>
kubectl scale --replicas=0 deployment axaservices-amq -n<namespace>
kubectl scale --replicas=0 deployment dxi-postgresql -n<namespace>


2) Verify that all pods have been terminated

kubectl get pods -n<namespace> | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-scheduler|axaservices-readserver|axaservices-amq|dxi-postgresql"


3) Scale up deployments in the following order 1 by 1.

IMPORTANT: Make sure all pods are start correctly by checking the pods logs using : kubectl logs -f <pod-name> -n <namespace>

as below:

kubectl scale --replicas=1 deployment dxi-postgresql -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:



kubectl scale --replicas=1 deployment axaservices-amq -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:

 

kubectl scale --replicas=1 deployment axaservices-readserver -n<namespace>

The start of this process can take several minutes


NOTE: You can ignore the below ERRORS:

[EMMCacheRefresher,mdo-serverCacheRefreshCheck] ERROR [] - TID[14015724] 5271596: JMS Request processing timeout, please wait for background processing
com.ca.emm.corejsvr.ExceptionWithNC: 5271596: JMS Request processing timeout, please wait for background processing

..

2021-11-30 14:03:09,521 [EMMCacheRefresher,mdo-serverCacheRefreshCheck] ERROR [] - TID[14015724] 3011599: Internal Error: Unable to complete this cache refresh cycle: 5271596: JMS Request processing timeout, please wait for background processing

kubectl scale --replicas=1 deployment axaservices-scheduler -n<namespace>     (If AXA has been installed)

Wait for below lines to appear in the pod log before starting the next pod:



kubectl scale --replicas=1 deployment dxi-readserver -n<namespace>

The start of this process can take several minutes

Wait for below lines to appear in the pod log before starting the next pod:



kubectl scale --replicas=1 deployment dxi-adminui -n<namespace>

Wait for below lines to appear in the pod log before starting the next pod:



kubectl scale --replicas=1 deployment apmservices-manager-001 -n<namespace>

Wait for below lines to appear before starting the next pod:



kubectl scale --replicas=1 deployment doireadserver -n<namespace>

The start of this process can take several minutes

Wait for below lines to appear in the pod log before starting the next pod:


NOTE: You will need to scroll up to find the above "Server startup"entry

 

kubectl scale --replicas=1 deployment doi-adminui -n<namespace>

Confirm you can see below lines in the pod log:


kubectl scale --replicas=1 deployment doi-automic-integration -n<namespace>

Confirm you can see below lines in the pod log:

3) Finally verify that all pods are up and running:

kubectl get pods -n<namespace> | egrep "doi-automic-integration|doi-adminui|doireadserver|apmservices-manager-001|dxi-adminui|dxi-readserver|axaservices-scheduler|axaservices-readserver|axaservices-amq|dxi-postgresql"

For example:

4) Login to DX UI




For DOI 1.3.2:

1) Scale down the following deployments:

kubectl scale --replicas=0 dc adminui -n<namespace>
kubectl scale --replicas=0 dc readserver -n<namespace>
kubectl scale --replicas=0 dc doireadserver -n<namespace>
kubectl scale --replicas=0 dc amq -n<namespace>
kubectl scale --replicas=0 dc postgres -n<namespace>

2) Scale up deployments in the following order 1 by 1
NOTE:  ensure all pods are starting correctly by checking the pods logs using : kubectl logs -f <pod-name> -n <namespace>

kubectl scale --replicas=1 dc doi-postgres -n<namespace>
kubectl scale --replicas=1 dc amq -n<namespace>
kubectl scale --replicas=1 dc doireadserver -n<namespace>
kubectl scale --replicas=1 dc readserver -n<namespace>
kubectl scale --replicas=1 dc adminui -n<namespace>

3) Verify that all pods are up and running

4) Login to DOI UI

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Attachments