DX O2 ServiceNow Integration Troubleshooting

Products

DX Operational Observability

Issue/Introduction

The following is a high-list of techniques and suggestions to employ when troubleshooting related to ServiceNow integration

A) Checklist

B) What files should I collect for Broadcom Support?

Environment

DX O2

Resolution

DATA FLOW:

A) CHECKLIST

STEP #1 : Check that all relevant pods and services are up and running
STEP #2 : Check ServiceNow configuration
STEP #3 : You can start by checking that manual ticket creation is working as expected troubleshooting
STEP #4 : Check Jarvis (kafka, zookeeper, elastic) health
STEP #5 : Check Alarm in ElasticSearch indices
STEP #6 : Check Incidentmanagement log
STEP #8 : Check Nim integration
STEP #8 : Check doireadserver for Incident display issues

STEP #1 : Check that all relevant pods and services are up and running

To integrate with ServiceNow, ensure that the following pods are running

• doi-incidentmanagement
• doi-nim
• doi-integrationgateway
• doi-tenantmanagement
• doireadserver
• axaservices-notifier-filter

axaservices-notify-filter-756ddc876f-2v7km 1/1 Running 0 14d
doi-incidentmanagement-56b66b6bff-5qtqv 1/1 Running 0 14d
doi-incidentmanagementpollingengine-7cf4bcfb4f-v92wl 1/1 Running 0 14d
doi-integrationgateway-7df8cd5d94-xvft8 1/1 Running 0 6d
doi-nim-7df49f5f89-vb7bq 1/1 Running 0 14d
doi-tenantmanagement-5fb7c64b96-cxj8t 1/1 Running 0 14d
doireadserver-6cb959d664-lv42v 1/1 Running 0 14d

STEP #2 : Check ServiceNow configuration

1) Go to Settings | Channels | <your ServiceNow configuration>

You can use the TEST button to verify connectivity

If you get an error:

a) Check ServiceNow connectivity, it could a firewall issue.
b) Use the Browser Developer Tool to further diagnose the issue.
c) Check NIM to ServiceNow connectivity, see:

DX OI Nim Troubleshooting

2) Go to Settings | Policies | <your Policy configuration>

Ensure the policy filter matches the service/raw alarm.

STEP #3 : You can start by checking that manual ticket creation is working as expected troubleshooting

Use the Browser Developer Tool for troubleshooting. Check if you see any error or exception

..

STEP #4 : Check Jarvis (kafka, zookeeper, elastic) health

If ServiceNow incident creation is still not working, check for a possible issue with Jarvis (kafka, zookeeper, elastic)

DX O2 Jarvis / Kafka Troubleshooting, Common Issues and Best Practices

STEP #5 : Check Alarm in ElasticSearch indices

1) Check the Alarm ingestion:

Verify the produce sources alarm appears in the Alarm product index: ao_itoa_alarms_<product>

For example, see to validate an UIM alarm:

{ES-Endpoint}/*alarms_uim*/_search?pretty&size=100&sort=timestamp:desc&q=nimid:<alarm-ID>

For more examples, see :DX O2 ElasticSearch Queries

2) Check the SNOW Incidents creation:

Verify the alarm appears in the following elastic indices in the below order:

For Raw Alarms: ao_itoa_alarms_all -> ao_itoa_channels -> ao_itoa_incidents
For Situation Alarms : ao_itoa_alarms_service_sa -> ao_itoa_channels -> ao_itoa_incidents

Below an example illustrating the troubleshooting process:

a) Check the alarm appears in "alarms_all" index

http://<elastic-endpoint>/*alarms_all*/_search?pretty&q=alarm_unique_id:<your alarm id>

b) Check the alarm appears in the "channels" index :

Check the channels attribute is added with <channelName>#OnPrem_ITSM

c) Check if the alarm appears in the "incident" index:

http://<elastic-endpoint>/*incidents*/_search?pretty&q=alarm_unique_id:<your alarm>

e) Check "alarms_all" index is updated with the ServiceNow incident number.

STEP #6 : Check Incidentmanagement log

Go to <NFS-dix>/OI/normalized-alarm/logs/<doi-normalized-alarm-pod>

Search for "troubleTicket" or "AlarmNotificationHandler", here is an example of the kind of output you should see:

STEP #8 : Check Nim integration

DX O2 Nim Troubleshooting

STEP #8 : Check doireadserver for Incident display issues

DX 02 Alarms do not display the generated ServiceNow incidents

B) WHAT FILES SHOULD I COLLECT FOR BROADCOM SUPPORT?

If you still need assistance, contact Broadcom Support (https://support.broadcom.com/) and provide the below information:

a) Details of the problematic alarm(s) (if possible, provide screenshots)

b) Result from Developer tool > Network tab : when creating a ServiceNow Incident manually:

c) Output of:

d) From ElasticSearch

Collect the result of the below queries:

Elastic Health:
http(s)://<ELASTIC_URL>/_cluster/health?pretty&human
http(s)://<ELASTIC_URL>/_nodes/stats/fs?pretty
http(s)://<ELASTIC_URL>/_nodes/stats/indices?pretty
http(s)://<ELASTIC_URL>/_cat/health?v
http(s)://<ELASTIC_URL>/_cat/nodes?v

Elastic indices:
http(s)://{ELASTIC_URL}/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
http(s)://<ELASTIC_URL>/*alarms_all*/_search?pretty&sort=@timestamp:desc&size=200
http(s)://<ELASTIC_URL>/*alarms_service_sa*/_search?pretty&sort=@timestamp:desc&size=200
http(s)://<ELASTIC_URL>/*channels*/_search?pretty&sort=@timestamp:desc&size=200
http(s)://<ELASTIC_URL>/*incidents*/_search?pretty&sort=@timestamp:desc&size=200

e) Logs:

doi-incidentmanagement service:
<NFS>/doiservices/incidentmanagement/<doi-incidentmanagement-pod>/incidentmanger.log
<NFS>/doiservices/incidentmanagement/restservices/restservices.log

doi-incidentmanagementpollingengine service: (e.g. poll updates SNOW to OI)
<NFS>/doiservices/incidentmanagementpollingengine/logs/itsm_pollingengine.log

Nim service:
kubectl cp <doi-nim-pod>:webapps/ca-nim-sm/WEB-INF/logs/Nim.log /tmp

doireadserver service:
- kubectl logs <doireadserver-pod>
- <NFS>/doiservices/readserver/logs/ca-doi-server-log.txt

doi-integrationgateway service:
- kubectl logs <doi-integrationgateway-pod>
- <NFS>/doiservices/integrationgateway/logs/integrationGateway.log

axaservices-notify-filter service
- kubectl logs <axaservices-notify-filter-pod>

f) Jarvis logs and evidences, see: DX O2 Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting