DX APM - Unable to login to tenant and mastertenant - jarvis-elasticsearch pods evicted

Products

• CA App Experience Analytics
• CA Application Performance Management (APM / Wily / Introscope)
• DX Application Performance Management

Issue/Introduction

Unable to login to APM 11 login UI, apmservices-gateway, dxportal, tenant, axa admin or masteradmin console


"Failed to login. Verify that user ID and password are correct"


Symptoms:
Evicted  jarvis-elasticsearch pods

Cause

Authorization is controlled by AXA services, the message indicates and there is some issue with either axa services, postgres data or elastic nodes.

In this case, the problem is caused by a lack of disk space in one of the Elastic Node


Below message is found when running kubectl describe <pod>

kubectl describe po jarvis-elasticsearch-3-7984bc48f9-zzp2j -n dxi
Name:           jarvis-elasticsearch-3-7984bc48f9-zzp2j
Namespace:      dxi
Node:           node5/
Start Time:     Fri, 28 Feb 2020 05:17:36 +0000
Labels:         app=jarvis-elasticsearch-3
                app-svc=jarvis-elasticsearch
                pod-template-hash=3540670495
                product_id=jarvis
                type=core
Annotations:    <none>
Status:         Failed
Reason:         Evicted
Message:        Pod The node was low on resource: [DiskPressure].
IP:
Controlled By:  ReplicaSet/jarvis-elasticsearch-3-7984bc48f9
..


Elastic node was configured with 200G only

As per documentation
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/1-0/installing/sizing-recommendations.html#concept.dita_9b35cd55dbc2557dd8739a25e94219ccf1842563_SizingProcess1

Elastic Nodes requires 500 GB SSD

Environment

APM 11.x, 19.4.x

Resolution

1. Increase disk space on Elastic nodes as per sizing recommendation

2. Deleted the evicted pods using 
kubectl delete pod <pod-name> -n<namespace>

3. Restart axaservices-adminui pod again

kubectl delete pod <axaservices-adminui pod-name> -n<namespace>