AIOps - ElasticSearch Queries
search cancel

AIOps - ElasticSearch Queries

book

Article ID: 207215

calendar_today

Updated On:

Products

DX Operational Intelligence DX Application Performance Management CA App Experience Analytics

Issue/Introduction

The following is a high list of queries to employ when troubleshooting performance, display and configuration issues related to ElasticSearch, AIOPs, OI, AXA and APM.

Environment

DX Platform 2x

** This KB is valid for DX On Premise version only, if you are using DX SaaS, contact Broadcom Support for assistance **

 

Resolution

 
- Connect to any kafka pod

  kubectl exec -ti <jarvis-kafka-pod> sh -n<namepsace>
 
- Query elastic database, for example:
 
   curl -XGET 'http://jarvis-elasticsearch-lb:9200/_cluster/health?pretty&human' |sort
 
 
NOTE: Jarvis APIS and ElasticSearch route/ingress endpoints are not longer available, however, you can re-create them as explained in https://knowledge.broadcom.com/external/article/226870
 
 
 

 

  Description Syntax
Elastic Health    
  Check Elastic version http(s)://<ELASTIC_URL>/
  Check Elastic Status (make sure status" : "green") http(s)://<ELASTIC_URL>/_cluster/health?pretty&human
  Check disk space usage http(s)://<ELASTIC_URL>/_nodes/stats/fs?pretty
  Node stats http(s)://<ELASTIC_URL>/_nodes/stats/indices?pretty
  ES health (check status is in green) http(s)://<ELASTIC_URL>/_cat/health?v
  Displays nodes in cluster (check memory, cpu, load) http(s)://<ELASTIC_URL>/_cat/nodes?v
  Check for possible errors during allocation, to get explanation on cluster issues http(s)://<ELASTIC_URL>/_cluster/allocation/explain?pretty
  reason for unassigned shards http(s)://<ELASTIC_URL>/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details&v
  To look for bulk rejection http(s)://<ELASTIC_URL>/_cat/thread_pool?v
  Identify backup folder http(s)://<ELASTIC_URL>/_snapshot/_all?pretty
  Shards details http(s)://<ELASTIC_URL>/_cat/shards?v
     
  Check disk space usage from Elastic servers Go to elastic node(s)
  - via server: /dxi/jarvis/elasticsearch/nodes/0/indices
  - via pod | terminal /var/data/elasticsearch/nodes/0/indices
  Execute: du -m --max-depth 1 | sort -rn | more
     
Maintenance    
  IMPORTANT:  
 
Avoid deleting indices. If required, delete indices from product indices ONLY, ie: alarms_apm, alarms_uim, alarms_anomaly
 
 
See: How to reduce data in Elastic indices: https://knowledge.broadcom.com/external/article/188786
 
 
If you have questions, contact Broadcom Support
 
     
  To delete indices curl -X DELETE http://<ELASTIC_URL>/<index-name>
     
Incides    
  List all indices http(s)://<ELASTIC_URL>/_cat/indices?v
  List indices by size: http(s)://<ELASTIC_URL>/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
  List indices by creation date http(s)://<ELASTIC_URL>/_cat/indices/?v&s=cds:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
  Memory usage of indices http://<ELASTIC_URL>/_cat/indices?v&h=i,tm&s=tm:desc
  to get settings of index http://<ELASTIC_URL>/<index-name>/_settings
  check Mapping http://<ELASTIC_URL>/_all/_mapping?pretty
  metadata for given index http://<ELASTIC_URL>/_cluster/state/metadata/<indexname>?pretty
     
Tenants    
  List tenant details http(s)://<ELASTIC_URL>/ao_dxi_tenants_1_1/_search?size=200&pretty
  List tenant details (legacy) http(s)://<ELASTIC_URL>/ao_tenants_1_1/_search?size=200&pretty
     
Jarvis Metadata    
  Query jarvis_metadata http(s)://<ELASTIC_URL>/jarvis_metadata/_search?pretty&q=md_type:"product_info"
     
Alarms_all    
  List past 200 alarms http(s)://<ELASTIC_URL>/*alarms_all*/_search?pretty&sort=@timestamp:desc&size=200
  Query for a specific Alarm id in "alarms_all* indices http(s)://<ELASTIC_URL>/*alarms_all*/_search?pretty&q=alarm_unique_id:<ALARM-ID>
  Query for specific alarm type http(s)://<ELASTIC_URL>/ao_itoa_alarms_all_1_1*/serch?pretty&q=alarmType:Prediction
     
UIM indices    
  List all uim indices http(s)://<ELASTIC_URL>/_cat/indices/*uim*?v
  List uim ci inventory http(s)://<ELASTIC_URL>/*inventory_uim_ci*/_search?sort=@timestamp:desc&size=200&pretty
  List uim device inventory http(s)://<ELASTIC_URL>/*inventory_uim_device*/_search?sort=@timestamp:desc&size=200&pretty
  List uim alarms http(s)://<ELASTIC_URL>/*alarms_uim*/_search?sort=@timestamp:desc&size=200&pretty
  Query for a specific Alarm id in *alarms_uim* incides http(s)://<ELASTIC_URL>/*alarms_uim*/_search?pretty&q=nimid:<ALARM-ID>
     
Spectrum indices    
  List all spectrum indices http(s)://<ELASTIC_URL>/_cat/indices/*spectrum*?v
  List spectrum alarms http(s)://<ELASTIC_URL>/*alarms_spectrum*/_search?sort=@timestamp:desc&size=200&pretty
  List spectrum inventory http(s)://<ELASTIC_URL>/*inventory_spectrum*/_search?sort=@timestamp:desc&size=200&pretty
     
NetOps / capm indices    
  List all capm incides http(s)://<ELASTIC_URL>/_cat/indices/*capm*?v
  List capm events http(s)://<ELASTIC_URL>/*events_capm*/_search?sort=@timestamp:desc&size=200&pretty
  List capm groups http(s)://<ELASTIC_URL>/*groups_capm*/_search?sort=@timestamp:desc&size=200&pretty
     
APM indices    
  List all apmindices http(s)://<ELASTIC_URL>/_cat/indices/*_apm*?v
  List apm inventory inventory http(s)://<ELASTIC_URL>/*inventory_apm*/_search?sort=@timestamp:desc&size=200&pretty
  List apm alarms inventory http(s)://<ELASTIC_URL>/*alarms_apm*/_search?sort=@timestamp:desc&size=200&pretty
  List apm tt http(s)://<ELASTIC_URL>/*apm_tt*/_search?pretty
     
LogAnalytics    
  all log incides http(s)://<ELASTIC_URL>/_cat/indices/*log*?v
  List all messages in syslog index (rsyslog and syslog-ng) http(s)://<ELASTIC_URL>/*logs_syslog*/_search?sort=@timestamp:desc&size=200&pretty
  List all messages in syslog index (nxlog) http(s)://<ELASTIC_URL>/*logs_eventlog*/_search?sort=@timestamp:desc&size=200&pretty
  List all messages in IIS log http(s)://<ELASTIC_URL>/*logs_iis*/_search?sort=@timestamp:desc&size=200&pretty
     
Situation    
  List all situations http(s)://<ELASTIC_URL>/*alarms_situation_cluster*/_search?pretty&size=100&sort=@timestamp:desc
  List all situations for a specific tenant http(s)://<ELASTIC_URL>/*alarms_situation_cluster*/_search?pretty&size=100&sort=@timestamp:desc&q=@tenant_id:<TENANT_ID>
     
ServiceNow    
  Data Flow:  
 
1) For Raw alarm : alarms are pushed to : alarms_all -> channels -> incidents
 
 
2) For Situation alarm: alarms are pushed to : alarms_all -> service_sa -> channels -> incidents
 
     
  List alarms services indices http://<ELASTIC_URL>/_cat/indices/*alarms_service_sa*?v
  Query for a specific Alarm id in "service_sa* indices http(s)://<ELASTIC_URL>/*alarms_service_sa*/_search?pretty&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>
  List channel indices http://<ELASTIC_URL>/_cat/indices/*channels*?v
  Query for a specific Alarm id in "channels* indices http(s)://<ELASTIC_URL>/*channels*/_search?pretty&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>
  List alarms services indices http://<ELASTIC_URL>/_cat/indices/*alarms_service_sa*?v
  Query for a specific Alarm id in "incidents* indices http(s)://<ELASTIC_URL>/*incidents*/_search?pretty&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>
     
Capacity Analytics / Predictive Insight    
  List all PI indices http(s)://<ELASTIC_URL>/_cat/indices/*pre*?v
  List uim PI alarms http(s)://<ELASTIC_URL>/*alarms_prediction*/_search?sort=@timestamp:desc&size=200&pretty
  List uim PI metrics http(s)://<ELASTIC_URL>/*alarms_prediction*/_search?sort=@timestamp:desc&size=200&pretty

 

Additional Information

ElasticSearch documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/7.x/index.html

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815