Simulating elasticsearch integration with NetOps PM to retrieve syslog messages

Products

• Network Observability
• CA Performance Management

Issue/Introduction

This article covers the sequence of steps to simulate the integration with Elastic to Retrieve Syslog messages

Environment

All Supported NetOps PM releases

Resolution

Assumptions: 

Elasticsearch service has been setup with port 9200 and is using HTTPS with Authorization Bearer token. This may vary in environment. 

• Create an Elasticsearch index as follows:

curl -X PUT "https://<ES Server>:9200/<ES Index>/" -H "Authorization: Bearer <AUTH TOKEN>" -H 'Content-Type: application/json' -d '
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "elastic.host": { "type": "keyword" },
      "@timestamp": { "type": "date" },
      "elastic.severity": { "type": "text" },
      "elastic.facility": { "type": "text" },
      "elastic.message": { "type": "text" }
    }
  }
}'

• [Optional] Set the result window to avoid shards related errors:

curl -X PUT "http://<ES Server>/<ES Index>/_settings" -H "Authorization: Bearer <AUTH TOKEN>" -H 'Content-Type: application/json' -d '
{
"index" : 
{"max_result_window" : 2100000}
}'

• Push messages on-to elasticsearcg index as follows:
  Example of 5 messages pushed accross multiple devices as follows:

curl -XPOST "http://<ES Server>:9200/<ES Index>/_doc/" -H 'Content-Type: application/json' -d'
{
  "elastic.host": "Switch",
  "@timestamp": "<TIMESTAMP>",
  "elastic.severity": "HIGH",
  "elastic.facility": "SYSLOG",
  "elastic.message": "Pushing 1st message into Elasticsearch."
}'

curl -XPOST "http://<ES Server>:9200/<ES Index>/_doc/" -H 'Content-Type: application/json' -d'
{
  "elastic.host": "Server 2",
  "@timestamp": "<TIMESTAMP>",
  "elastic.severity": "HIGH",
  "elastic.facility": "SYSLOG",
  "elastic.message": "Pushing 2nd message into Elasticsearch."
}'

curl -XPOST "http://<ES Server>:9200/<ES Index>/_doc/" -H 'Content-Type: application/json' -d'
{
  "elastic.host": "Server 3",
  "@timestamp": "<TIMESTAMP>",
  "elastic.severity": "HIGH",
  "elastic.facility": "SYSLOG",
  "elastic.message": "Pushing 3rd message into Elasticsearch."
}'

curl -XPOST "http://<ES Server>:9200/<ES Index>/_doc/" -H 'Content-Type: application/json' -d'
{
  "elastic.host": "Server 4",
  "@timestamp": "<TIMESTAMP>",
  "elastic.severity": "HIGH",
  "elastic.facility": "SYSLOG",
  "elastic.message": "Pushing 4th message into Elasticsearch."
}'

curl -XPOST "http://<ES Server>:9200/<ES Index>/_doc/" -H 'Content-Type: application/json' -d'
{
  "elastic.host": "Switch",
  "@timestamp": "<TIMESTAMP>",
  "elastic.severity": "HIGH",
  "elastic.facility": "SYSLOG",
  "elastic.message": "Pushing 5th message into Elasticsearch."
}'

• Configure the syslog connector on the Portal as explained under : Integrate with Elastic to Retrieve Syslog
  Example: (where "es" is a sample index)
  
  
• Browse any device with valid syslog entry on ES in the inventory and check the Log Events tab
  
• You can see the messages mapped with each device accordingly in the descending order:

Additional Information

You will only see syslog as alarms, if those syslog entries are sent to spectrum via rsyslog and turned into alarms, and the device is synched to PM from Spectrum.
For events that are synched from elasticsearch, the elastic-query will be executed every-time a user visits the Log Events tab and hence they are visible only at the Log Events level.