ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

AIOps - ElasticSearch Queries

book

Article ID: 207215

calendar_today

Updated On:

Products

DX Operational Intelligence DX Application Performance Management CA App Experience Analytics

Issue/Introduction

The following is a high list of queries to employ when troubleshooting performance, display and configuration issues related to ElasticSearch, AIOPs, OI, AXA and APM.

Environment

DX Operational Intelligence 20.2.x and higher versions
DX Application Performance Management 20.2.x and higher version

** This KB is valid for DX On Premise version only, if you are using DX SaaS, contact Broadcom Support for assistance **

 

Resolution

If 21.3.x
 
- Connect to any kafka pod

  kubectl exec -ti <jarvis-kafka-pod> sh -n<namepsace>
 
- Query elastic database, for example:
 
   curl -XGET 'http://jarvis-elasticsearch-lb:9200/_cluster/health?pretty&human' |sort
 
 
NOTE: Jarvis APIS and ElasticSearch route/ingress endpoints are not longer available, however, you can re-create them as explained in https://knowledge.broadcom.com/external/article/226870
 
 
If 20.2.x
 

Find out the ElasticSearch endpoint:

If Kubernetes:  kubectl get ingress -n<dxi-namespace> | grep jarvis-es

for example:

kubectl get ingress -ndxi | grep jarvis
jarvis-es                             <none>   es.10.109.32.88.nip.io             10.109.32.88   80      19d

If Openshift:     oc get routes -n<dxi-namespace> | grep jarvis-es

for example:

oc get routes -ndxi | grep jarvis
jarvis-es-7krrv                             es.munqa001493.bpc.broadcom.net                /                  jarvis-elasticsearch-lb       9200                                          None

 

  Description Syntax Example
Elastic Health      
  Check Elastic version http(s)://<ELASTIC_URL>/ http://es.munqa001493.bpc.broadcom.net
  Check Elastic Status (make sure status" : "green") http(s)://<ELASTIC_URL>/_cluster/health?pretty&human http://es.munqa001493.bpc.broadcom.net/_cluster/health?pretty&human
  Check disk space usage http(s)://<ELASTIC_URL>/_nodes/stats/fs?pretty http://es.munqa001493.bpc.broadcom.net/_nodes/stats/fs?pretty
  Node stats http(s)://<ELASTIC_URL>/_nodes/stats/indices?pretty http://es.munqa001493.bpc.broadcom.net/_nodes/stats/indices?pretty
  ES health (check status is in green) http(s)://<ELASTIC_URL>/_cat/health?v http://es.munqa001493.bpc.broadcom.net/_cat/health?v
  Displays nodes in cluster (check memory, cpu, load) http(s)://<ELASTIC_URL>/_cat/nodes?v http://es.munqa001493.bpc.broadcom.net/_cat/nodes?v
  Check for possible errors during allocation, to get explanation on cluster issues http(s)://<ELASTIC_URL>/_cluster/allocation/explain?pretty http://es.munqa001493.bpc.broadcom.net/_cluster/allocation/explain?pretty
  reason for unassigned shards http(s)://<ELASTIC_URL>/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details&v http://es.munqa001493.bpc.broadcom.net/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details&v
  To look for bulk rejection http(s)://<ELASTIC_URL>/_cat/thread_pool?v http://es.munqa001493.bpc.broadcom.net/_cat/thread_pool?v
  Identify backup folder http(s)://<ELASTIC_URL>/_snapshot/_all?pretty http://es.munqa001493.bpc.broadcom.net/_snapshot/_all?pretty
  Shards details http(s)://<ELASTIC_URL>/_cat/shards?v http://es.munqa001493.bpc.broadcom.net/_cat/shards?v
       
  Check disk space usage from Elastic servers Go to elastic node(s)  
  - via server: /dxi/jarvis/elasticsearch/nodes/0/indices  
  - via pod | terminal /var/data/elasticsearch/nodes/0/indices  
  Execute: du -m --max-depth 1 | sort -rn | more  
       
Maintenance      
  IMPORTANT:    
 
Avoid deleting indices. If required, delete indices from product indices ONLY, ie: alarms_apm, alarms_uim, alarms_anomaly
   
 
See: How to reduce data in Elastic indices: https://knowledge.broadcom.com/external/article/188786
   
 
If you have questions, contact Broadcom Support
   
       
  To delete indices curl -X DELETE http://<ELASTIC_URL>/<index-name> curl -X DELETE http://es.munqa001493.bpc.broadcom.net/ao_itoa_logs_syslog_1_25
       
Incides      
  List all indices http(s)://<ELASTIC_URL>/_cat/indices?v http://es.munqa001493.bpc.broadcom.net/_cat/indices?v
  List indices by size: http(s)://<ELASTIC_URL>/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds http://es.munqa001493.bpc.broadcom.net/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
  List indices by creation date http(s)://<ELASTIC_URL>/_cat/indices/?v&s=cds:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds http://es.munqa001493.bpc.broadcom.net/_cat/indices/?v&s=cds:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
  Memory usage of indices http://<ELASTIC_URL>/_cat/indices?v&h=i,tm&s=tm:desc http://es.munqa001493.bpc.broadcom.net/_cat/indices?v&h=i,tm&s=tm:desc
  to get settings of index http://<ELASTIC_URL>/<index-name>/_settings http://es.munqa001493.bpc.broadcom.net/*apm_tt*/_settings
  check Mapping http://<ELASTIC_URL>/_all/_mapping?pretty http://es.munqa001493.bpc.broadcom.net/_all/_mapping?pretty
  metadata for given index http://<ELASTIC_URL>/_cluster/state/metadata/<indexname>?pretty http://es.munqa001493.bpc.broadcom.net/_cluster/state/metadata/ao_apm_tt_2_1?pretty
       
Tenants      
  List tenant details http(s)://<ELASTIC_URL>/ao_dxi_tenants_1_1/_search?size=200&pretty http://es.munqa001493.bpc.broadcom.net/ao_dxi_tenants_1_1/_search?size=200&pretty
  List tenant details (legacy) http(s)://<ELASTIC_URL>/ao_tenants_1_1/_search?size=200&pretty http://es.munqa001493.bpc.broadcom.net/ao_tenants_1_1/_search?size=200&pretty
       
Jarvis Metadata      
  Query jarvis_metadata http(s)://<ELASTIC_URL>/jarvis_metadata/_search?pretty&q=md_type:"product_info" http://es.munqa001493.bpc.broadcom.net/jarvis_metadata/_search?pretty&q=md_type:"product_info"
       
Alarms_all      
  List past 200 alarms http(s)://<ELASTIC_URL>/*alarms_all*/_search?pretty&[email protected]:desc&size=200 http://es.munqa001493.bpc.broadcom.net/*alarms_all*/_search?pretty&[email protected]:desc&size=200
  Query for a specific Alarm id in "alarms_all* indices http(s)://<ELASTIC_URL>/*alarms_all*/_search?pretty&q=alarm_unique_id:<ALARM-ID> http://es.munqa001493.bpc.broadcom.net/*alarms_all*/_search?pretty&q=alarm_unique_id:QV93808167-75242
  Query for specific alarm type http(s)://<ELASTIC_URL>/ao_itoa_alarms_all_1_1*/serch?pretty&q=alarmType:Prediction http://es.munqa001493.bpc.broadcom.net/ao_itoa_alarms_all_1_1*/serch?pretty&q=alarmType:Prediction
       
UIM indices      
  List all uim indices http(s)://<ELASTIC_URL>/_cat/indices/*uim*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*uim*?v
  List uim ci inventory http(s)://<ELASTIC_URL>/*inventory_uim_ci*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*inventory_uim_ci*/[email protected]:desc&size=200&pretty
  List uim device inventory http(s)://<ELASTIC_URL>/*inventory_uim_device*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*inventory_uim_device*/[email protected]:desc&size=200&pretty
  List uim alarms http(s)://<ELASTIC_URL>/*alarms_uim*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*alarms_uim*/[email protected]:desc&size=200&pretty
  Query for a specific Alarm id in *alarms_uim* incides http(s)://<ELASTIC_URL>/*alarms_uim*/_search?pretty&q=nimid:<ALARM-ID> http://es.munqa001493.bpc.broadcom.net/*alarms_uim*/_search?pretty&q=nimid:PG85096332-05512
       
Spectrum indices      
  List all spectrum indices http(s)://<ELASTIC_URL>/_cat/indices/*spectrum*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*spectrum*?v
  List spectrum alarms http(s)://<ELASTIC_URL>/*alarms_spectrum*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*alarms_spectrum*/[email protected]:desc&size=200&pretty
  List spectrum inventory http(s)://<ELASTIC_URL>/*inventory_spectrum*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*inventory_spectrum*/[email protected]:desc&size=200&pretty
       
NetOps / capm indices      
  List all capm incides http(s)://<ELASTIC_URL>/_cat/indices/*capm*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*capm*?v
  List capm events http(s)://<ELASTIC_URL>/*events_capm*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*events_capm*/[email protected]:desc&size=200&pretty
  List capm groups http(s)://<ELASTIC_URL>/*groups_capm*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*groups_capm*/[email protected]:desc&size=200&pretty
       
APM indices      
  List all apmindices http(s)://<ELASTIC_URL>/_cat/indices/*_apm*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*_apm*?v
  List apm inventory inventory http(s)://<ELASTIC_URL>/*inventory_apm*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*inventory_apm*/[email protected]:desc&size=200&pretty
  List apm alarms inventory http(s)://<ELASTIC_URL>/*alarms_apm*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*alarms_apm*/[email protected]:desc&size=200&pretty
  List apm tt http(s)://<ELASTIC_URL>/*apm_tt*/_search?pretty http://es.munqa001493.bpc.broadcom.net/*apm_tt*/_search?pretty
       
LogAnalytics      
  all log incides http(s)://<ELASTIC_URL>/_cat/indices/*log*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*log*?v
  List all messages in syslog index (rsyslog and syslog-ng) http(s)://<ELASTIC_URL>/*logs_syslog*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*logs_syslog*/[email protected]:desc&size=200&pretty
  List all messages in syslog index (nxlog) http(s)://<ELASTIC_URL>/*logs_eventlog*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*logs_eventlog*/[email protected]:desc&size=200&pretty
  List all messages in IIS log http(s)://<ELASTIC_URL>/*logs_iis*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*logs_iis*/[email protected]:desc&size=200&pretty
       
Situation      
  List all situations http(s)://<ELASTIC_URL>/*alarms_situation_cluster*/_search?pretty&size=100&[email protected]:desc http://es.munqa001493.bpc.broadcom.net/*alarms_situation_cluster*/_search?pretty&size=100&[email protected]:desc
  List all situations for a specific tenant http(s)://<ELASTIC_URL>/*alarms_situation_cluster*/_search?pretty&size=100&[email protected]:desc&[email protected]_id:<TENANT_ID> http://es.munqa001493.bpc.broadcom.net/*alarms_situation_cluster*/_search?pretty&size=100&[email protected]:desc&[email protected]_id:4E1F2D8E-AA5F-4A41-A1DF-D60F8C5E9B3A
       
ServiceNow      
  Data Flow:    
 
1) For Raw alarm : alarms are pushed to : alarms_all -> channels -> incidents
   
 
2) For Situation alarm: alarms are pushed to : alarms_all -> service_sa -> channels -> incidents
   
       
  List alarms services indices http://<ELASTIC_URL>/_cat/indices/*alarms_service_sa*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*alarms_service_sa*?v
  Query for a specific Alarm id in "service_sa* indices http(s)://<ELASTIC_URL>/*alarms_service_sa*/_search?pretty&[email protected]:desc&size=200&q=alarm_unique_id:<ALARM-ID> http://es.munqa001493.bpc.broadcom.net/*alarms_service_sa*/_search?pretty&[email protected]:desc&size=200&q=alarm_unique_id:QV93808167-75242
  List channel indices http://<ELASTIC_URL>/_cat/indices/*channels*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*channels*?v
  Query for a specific Alarm id in "channels* indices http(s)://<ELASTIC_URL>/*channels*/_search?pretty&[email protected]:desc&size=200&q=alarm_unique_id:<ALARM-ID> http://es.munqa001493.bpc.broadcom.net/*channels*/_search?pretty&[email protected]:desc&size=200&q=alarm_unique_id:QV93808167-75242
  List alarms services indices http://<ELASTIC_URL>/_cat/indices/*alarms_service_sa*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*alarms_service_sa*?v
  Query for a specific Alarm id in "incidents* indices http(s)://<ELASTIC_URL>/*incidents*/_search?pretty&[email protected]:desc&size=200&q=alarm_unique_id:<ALARM-ID> http://es.munqa001493.bpc.broadcom.net/*incidents*/_search?pretty&[email protected]:desc&size=200&q=alarm_unique_id:QV93808167-75242
       
Capacity Analytics / Predictive Insight      
  List all PI indices http(s)://<ELASTIC_URL>/_cat/indices/*pre*?v http://es.munqa001493.bpc.broadcom.net/_cat/indices/*pre*?v
  List uim PI alarms http(s)://<ELASTIC_URL>/*alarms_prediction*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*alarms_prediction*/[email protected]:desc&size=200&pretty
  List uim PI metrics http(s)://<ELASTIC_URL>/*alarms_prediction*/[email protected]:desc&size=200&pretty http://es.munqa001493.bpc.broadcom.net/*alarms_prediction*/[email protected]:desc&size=200&pretty

 

Additional Information

ElasticSearch documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/7.x/index.html

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Attachments