AIOps - ElasticSearch Queries

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

The following is a high list of queries to employ when troubleshooting performance, display and configuration issues related to ElasticSearch

Environment

DX Platform 23.x

 

IMPORTANT: ** This KB is valid for DX On Premise version only, if you are using DX SaaS, contact Broadcom Support for assistance **

 

Resolution

Table of Contents

• OPTION 1: How to query ElasticSearch from dxi pods
•  
• OPTION 2: How to query ElasticSearch from a Client Browser

 

OPTION 1: How to query ElasticSearch from dxi pods

 

By default, the Jarvis APIS and Elastic routes/ingress endpoints are only available inside the kubernetes or openshift cluster. Follow the below steps to query Elastic data:

 

STEP 1: Connect to an Elasticsearch or kafka pod:

 

kubectl exec -ti <dxi-pod-name> sh -n<dxi-namepsace>

 

 

STEP 2: Use the correct elastic endpoint for your queries:

 

For 23.1 : Elasticsearch endpoint is : jarvis-elasticsearch-lb

 

For 23.2 and higher versions use : elasticsearch-master

 

 

STEP 3: Query ES using CURL command:

 

Syntax: 

curl -XGET 'http://<ES_ENDPOINT>:9200/<ES_QUERY>' 

 

Where <ES_ENDPOINT> is your Elastic endpoint

 

 

Example how to query for ES health in 23.2+:

curl -XGET 'http://elasticsearch-master:9200/_cluster/health?pretty&human' 

 

Example how to query for ES health in 23.1

curl -XGET 'http://jarvis-elasticsearch-lb:9200/_cluster/health?pretty&human'

 

 

Below is a summary table list of some common ES queries:

	DESCRIPTION	QUERY
Elastic Health
	Check Elastic Status (make sure status" : "green")	curl -XGET 'http://<ES_ENDPOINT>:9200/_cluster/health?pretty&human'
	Check disk space usage	curl -XGET 'http://<ES_ENDPOINT>:9200/_nodes/stats/fs?pretty'
	Node stats	curl -XGET 'http://<ES_ENDPOINT>:9200/_nodes/stats/indices?pretty'
	ES health (check status is in green)	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/health?v'
	Displays nodes in cluster (check memory, cpu, load)	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/nodes?v'
	Check for possible errors during allocation, to get explanation on cluster issues	curl -XGET 'http://<ES_ENDPOINT>:9200/_cluster/allocation/explain?pretty'
	reason for unassigned shards	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details&v'
	To look for bulk rejection	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/thread_pool?v'
	Identify backup folder	curl -XGET 'http://<ES_ENDPOINT>:9200/_snapshot/_all?pretty'
	Shards details	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/shards?v'
	Check disk space usage from Elastic servers
	- option 1: go to each elastic server	cd <ES-home>/nodes/0/indices
	- option 2: go inside of the elastic pod(s)	/var/data/elasticsearch/nodes/0/indices
	- execute:	du -m --max-depth 1 | sort -rn | more
Delete Incides		IMPORTANT : - Contact Broadcom Support for assistance - If you need to reduce elastic disk usage, you can reduce its data retention, see: https://knowledge.broadcom.com/external/article/207161
	To delete indices	curl -X DELETE http://<ES_ENDPOINT>/<index-name>
Query Indices
	List all indices	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices?v'
	List indices by size:	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds'
	List indices by creation date	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices/?v&s=cds:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds'
	Memory usage of indices	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices?v&h=i,tm&s=tm:desc'
	to get settings of index	curl -XGET 'http://<ES_ENDPOINT>:9200/<index-name>/_settings'
	check Mapping	curl -XGET 'http://<ES_ENDPOINT>:9200/_all/_mapping?pretty'
	metadata for given index	curl -XGET 'http://<ES_ENDPOINT>:9200/_cluster/state/metadata/<indexname>?pretty'
Tenants
	List tenant details	curl -XGET 'http://<ES_ENDPOINT>:9200/ao_dxi_tenants_1_1/_search?size=200&pretty'
	List tenant details (legacy)	curl -XGET 'http://<ES_ENDPOINT>:9200/ao_tenants_1_1/_search?size=200&pretty'
Jarvis Metadata
	Query jarvis_metadata	curl -XGET 'http://<ES_ENDPOINT>:9200/jarvis_metadata/_search?pretty'&q=md_type:"product_info"'
Alarms_all
	List past 200 alarms	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_all*/_search?pretty'&sort=@timestamp:desc&size=200'
	Query for a specific Alarm id in "alarms_all* indices	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_all*/_search?pretty'&q=alarm_unique_id:<ALARM-ID>'
	Query for specific alarm type	curl -XGET 'http://<ES_ENDPOINT>:9200/ao_itoa_alarms_all_1_1*/serch?pretty'&q=alarmType:Prediction'
	Alarm details from alarms_all index for spectrum alarms	curl -XGET 'http://<ES_ENDPOINT>:9200/ao_itoa_alarms_all*/_search?q=alarm_unique_id:"spectrumAlarmId"&pretty'
UIM indices
	List all uim indices	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices/*uim*?v'
	List uim ci inventory	curl -XGET 'http://<ES_ENDPOINT>:9200/*inventory_uim_ci*/_search?sort=@timestamp:desc&size=200&pretty'
	List uim device inventory	curl -XGET 'http://<ES_ENDPOINT>:9200/*inventory_uim_device*/_search?sort=@timestamp:desc&size=200&pretty'
	List uim alarms	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_uim*/_search?sort=@timestamp:desc&size=200&pretty'
	Query for a specific Alarm id in *alarms_uim* incides	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_uim*/_search?pretty'&q=nimid:<ALARM-ID>'
Spectrum indices
	List all spectrum indices	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices/*spectrum*?v'
	List spectrum alarms	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_spectrum*/_search?sort=@timestamp:desc&size=200&pretty'
	List spectrum inventory	curl -XGET 'http://<ES_ENDPOINT>:9200/*inventory_spectrum*/_search?sort=@timestamp:desc&size=200&pretty'
	Alarm details from alarms_spectrum index	curl -XGET 'http://<ES_ENDPOINT>:9200/ao_itoa_alarms_spectrum*/_search?q=globalAlarmID:"spectrumAlarmId"&pretty'
NetOps / capm indices
	List all capm incides	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices/*capm*?v'
	List capm events	curl -XGET 'http://<ES_ENDPOINT>:9200/*events_capm*/_search?sort=@timestamp:desc&size=200&pretty'
	List capm groups	curl -XGET 'http://<ES_ENDPOINT>:9200/*groups_capm*/_search?sort=@timestamp:desc&size=200&pretty'
APM indices
	List all apmindices	curl -XGET 'http://<ES_ENDPOINT>:9200/_cat/indices/*_apm*?v'
	List apm inventory inventory	curl -XGET 'http://<ES_ENDPOINT>:9200/*inventory_apm*/_search?sort=@timestamp:desc&size=200&pretty'
	List apm alarms inventory	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_apm*/_search?sort=@timestamp:desc&size=200&pretty'
	List apm tt	curl -XGET 'http://<ES_ENDPOINT>:9200/*apm_tt*/_search?pretty'
Situation
	List all situations	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_situation_cluster*/_search?pretty'&size=100&sort=@timestamp:desc'
	List all situations for a specific tenant	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_situation_cluster*/_search?pretty'&size=100&sort=@timestamp:desc&q=@tenant_id:<TENANT_ID>'
ServiceNow
	List alarms services indices	http://<ES_ENDPOINT>/_cat/indices/*alarms_service_sa*?v'
	Query for a specific Alarm id in "service_sa* indices	curl -XGET 'http://<ES_ENDPOINT>:9200/*alarms_service_sa*/_search?pretty'&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>'
	List channel indices	http://<ES_ENDPOINT>/_cat/indices/*channels*?v'
	Query for a specific Alarm id in "channels* indices	curl -XGET 'http://<ES_ENDPOINT>:9200/*channels*/_search?pretty'&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>'
	List alarms services indices	http://<ES_ENDPOINT>/_cat/indices/*alarms_service_sa*?v'
	Query for a specific Alarm id in "incidents* indices	curl -XGET 'http://<ES_ENDPOINT>:9200/*incidents*/_search?pretty'&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>'

OPTION 2: How to query ElasticSearch from a Client Browser

You can make the Elastic Endpoint accessible from your browser as described here: https://knowledge.broadcom.com/external/article/226870

Once Elastic routes/ingress endpoints are available you will be able to run the below common queries:

To list DX Platform Routes and Ingresses you can use:

In kubernetes: kubectl get ingress -n <namespace>
In Openshift:    oc -n <namespace> get route   

	DESCRIPTION	QUERY
Elastic Health
	Check Elastic version	http(s)://<ES_ENDPOINT>/
	Check Elastic Status (make sure status" : "green")	http(s)://<ES_ENDPOINT>/_cluster/health?pretty&human
	Check disk space usage	http(s)://<ES_ENDPOINT>/_nodes/stats/fs?pretty
	Node stats	http(s)://<ES_ENDPOINT>/_nodes/stats/indices?pretty
	ES health (check status is in green)	http(s)://<ES_ENDPOINT>/_cat/health?v
	Displays nodes in cluster (check memory, cpu, load)	http(s)://<ES_ENDPOINT>/_cat/nodes?v
	Check for possible errors during allocation, to get explanation on cluster issues	http(s)://<ES_ENDPOINT>/_cluster/allocation/explain?pretty
	reason for unassigned shards	http(s)://<ES_ENDPOINT>/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details&v
	To look for bulk rejection	http(s)://<ES_ENDPOINT>/_cat/thread_pool?v
	Identify backup folder	http(s)://<ES_ENDPOINT>/_snapshot/_all?pretty
	Shards details	http(s)://<ES_ENDPOINT>/_cat/shards?v
	Check disk space usage from Elastic servers	Go to elastic node(s)
	- via server:	/dxi/jarvis/elasticsearch/nodes/0/indices
	- via pod | terminal	/var/data/elasticsearch/nodes/0/indices
	Execute:	du -m --max-depth 1 | sort -rn | more
Delete indices
	IMPORTANT:
	Avoid deleting indices. If required, delete indices from product indices ONLY, ie: alarms_apm, alarms_uim, alarms_anomaly
	See: How to reduce data in Elastic indices: https://knowledge.broadcom.com/external/article/188786
	If you have questions, contact Broadcom Support
	To delete indices	curl -X DELETE http://<ES_ENDPOINT>/<index-name>
Incides
	List all indices	http(s)://<ES_ENDPOINT>/_cat/indices?v
	List indices by size:	http(s)://<ES_ENDPOINT>/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
	List indices by creation date	http(s)://<ES_ENDPOINT>/_cat/indices/?v&s=cds:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
	Memory usage of indices	http://<ES_ENDPOINT>/_cat/indices?v&h=i,tm&s=tm:desc
	to get settings of index	http://<ES_ENDPOINT>/<index-name>/_settings
	check Mapping	http://<ES_ENDPOINT>/_all/_mapping?pretty
	metadata for given index	http://<ES_ENDPOINT>/_cluster/state/metadata/<indexname>?pretty
Tenants
	List tenant details	http(s)://<ES_ENDPOINT>/ao_dxi_tenants_1_1/_search?size=200&pretty
	List tenant details (legacy)	http(s)://<ES_ENDPOINT>/ao_tenants_1_1/_search?size=200&pretty
Jarvis Metadata
	Query jarvis_metadata	http(s)://<ES_ENDPOINT>/jarvis_metadata/_search?pretty&q=md_type:"product_info"
Alarms_all
	List past 200 alarms	http(s)://<ES_ENDPOINT>/*alarms_all*/_search?pretty&sort=@timestamp:desc&size=200
	Query for a specific Alarm id in "alarms_all* indices	http(s)://<ES_ENDPOINT>/*alarms_all*/_search?pretty&q=alarm_unique_id:<ALARM-ID>
	Query for specific alarm type	http(s)://<ES_ENDPOINT>/ao_itoa_alarms_all_1_1*/serch?pretty&q=alarmType:Prediction
	Alarm details from alarms_all index for spectrum alarms	http(s)://<ES_ENDPOINT>/ao_itoa_alarms_all*/_search?q=alarm_unique_id:"spectrumAlarmId"&pretty
UIM indices
	List all uim indices	http(s)://<ES_ENDPOINT>/_cat/indices/*uim*?v
	List uim ci inventory	http(s)://<ES_ENDPOINT>/*inventory_uim_ci*/_search?sort=@timestamp:desc&size=200&pretty
	List uim device inventory	http(s)://<ES_ENDPOINT>/*inventory_uim_device*/_search?sort=@timestamp:desc&size=200&pretty
	List uim alarms	http(s)://<ES_ENDPOINT>/*alarms_uim*/_search?sort=@timestamp:desc&size=200&pretty
	Query for a specific Alarm id in *alarms_uim* incides	http(s)://<ES_ENDPOINT>/*alarms_uim*/_search?pretty&q=nimid:<ALARM-ID>
Spectrum indices
	List all spectrum indices	http(s)://<ES_ENDPOINT>/_cat/indices/*spectrum*?v
	List spectrum alarms	http(s)://<ES_ENDPOINT>/*alarms_spectrum*/_search?sort=@timestamp:desc&size=200&pretty
	List spectrum inventory	http(s)://<ES_ENDPOINT>/*inventory_spectrum*/_search?sort=@timestamp:desc&size=200&pretty
	Alarm details from alarms_spectrum index	http(s)://<ES_ENDPOINT>/ao_itoa_alarms_spectrum*/_search?q=globalAlarmID:"spectrumAlarmId"&pretty
NetOps / capm indices
	List all capm incides	http(s)://<ES_ENDPOINT>/_cat/indices/*capm*?v
	List capm events	http(s)://<ES_ENDPOINT>/*events_capm*/_search?sort=@timestamp:desc&size=200&pretty
	List capm groups	http(s)://<ES_ENDPOINT>/*groups_capm*/_search?sort=@timestamp:desc&size=200&pretty
APM indices
	List all apmindices	http(s)://<ES_ENDPOINT>/_cat/indices/*_apm*?v
	List apm inventory inventory	http(s)://<ES_ENDPOINT>/*inventory_apm*/_search?sort=@timestamp:desc&size=200&pretty
	List apm alarms inventory	http(s)://<ES_ENDPOINT>/*alarms_apm*/_search?sort=@timestamp:desc&size=200&pretty
	List apm tt	http(s)://<ES_ENDPOINT>/*apm_tt*/_search?pretty
Situation
	List all situations	http(s)://<ES_ENDPOINT>/*alarms_situation_cluster*/_search?pretty&size=100&sort=@timestamp:desc
	List all situations for a specific tenant	http(s)://<ES_ENDPOINT>/*alarms_situation_cluster*/_search?pretty&size=100&sort=@timestamp:desc&q=@tenant_id:<TENANT_ID>
ServiceNow
	Data Flow:
	1) For Raw alarm : alarms are pushed to : alarms_all -> channels -> incidents
	2) For Situation alarm: alarms are pushed to : alarms_all -> service_sa -> channels -> incidents
	List alarms services indices	http://<ES_ENDPOINT>/_cat/indices/*alarms_service_sa*?v
	Query for a specific Alarm id in "service_sa* indices	http(s)://<ES_ENDPOINT>/*alarms_service_sa*/_search?pretty&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>
	List channel indices	http://<ES_ENDPOINT>/_cat/indices/*channels*?v
	Query for a specific Alarm id in "channels* indices	http(s)://<ES_ENDPOINT>/*channels*/_search?pretty&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>
	List alarms services indices	http://<ES_ENDPOINT>/_cat/indices/*alarms_service_sa*?v
	Query for a specific Alarm id in "incidents* indices	http(s)://<ES_ENDPOINT>/*incidents*/_search?pretty&sort=@timestamp:desc&size=200&q=alarm_unique_id:<ALARM-ID>

 

Additional Information

ElasticSearch documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/7.x/index.html

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815