Once in a while Service Now (SNOW) creates two tickets for the same alarm

Products

DX NetOps CA Spectrum

Issue/Introduction

The SD_notifier that is running the primary MLS is integrated with the OneClick server. There is no other instance of the SD_notifier running on any SpectroSERVER machine.

They have 5 OneClick servers. The SNOW integration is enabled on all of them. The reason behind this is that in case they need to create a ticket manually, they can do it from any OC behind a Load Balancer.

Even though the SNOW integration is enabled on all OC servers, no alarm type is selected to open a ticket automatically. And requested that no operator open a ticket manually.

Environment

Spectrum 22.2.x and Service Now integration

Cause

When OneClick/NIM creates an incident in NIM for some alarms (for example: Alarm ID: 65798420-8fce-1033-0115-00505697acab), the incident is being created in ServiceNow, however, there was an error returning from ServiceNow along with error code 500 to NIM like below:

Error executing business rule 'Fill CI for spectrum tickets'</faultstring><detail>ReferenceError: "cmdb" is not defined.

NIM will retry to create the incident if it doesn't get a 200 response from ServiceNow on the first attempt. As ServiceNow is returning a 500 error due to some ServiceNow rule failures shown above, NIM is sending the 2nd call and it is creating another incident for the same alarm id and getting the 500 return code again due to the same business rule failure (by this time ServiceNow has 2 incidents being created for the same alarm) , but after 2 calls NIM stopped retrying it as it reaches maximum tries. Due to which we are seeing two incidents for the same alarm.

Following is the ServiceNow business rule ERROR seen on NIM logs:

=========================================================

responseCode 500
The response is ::::::::::::: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode>SOAP-ENV:Server</faultcode><faultstring>Error executing business rule 'Fill CI for spectrum tickets'</faultstring><detail>ReferenceError: "cmdb" is not defined. (sys_script.154e085e1ba48d50c532da8ee54bcb1d.script; line 110)
at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:262)
at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:137)
at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:348)
at com.glide.script.BusinessRule.runScript(BusinessRule.java:443)
at com.glide.script.BusinessRule.runSyncBusinessRule(BusinessRule.java:324)
at com.glide.script.BusinessRule.runBusinessRule(BusinessRule.java:217)
at com.glide.script.BusinessRule.run(BusinessRule.java:193)
at com.glide.script.GlideRecordBusinessRules.run(GlideRecordBusinessRules.java:264)
at com.glide.script.GlideRecord.runScripts(GlideRecord.java:1452)
at com.glide.script.GlideRecord.runScriptsAndEngines(GlideRecord.java:1390)
at com.glide.script.GlideRecord.postChange(GlideRecord.java:5312)
at com.glide.script.GlideRecord.postInsert(GlideRecord.java:5276)
at com.glide.script.GlideRecord.insert(GlideRecord.java:5199)
at com.glide.script.GlideRecord.insert(GlideRecord.java:5091)
at com.glide.processors.soap.command.Insert.process(Insert.java:57)
at com.glide.processors.soap.SOAPProcessorThread.doCommand(SOAPProcessorThread.java:312)
at com.glide.processors.soap.SOAPProcessorThread.doCommand(SOAPProcessorThread.java:299)
at com.glide.processors.soap.SOAPProcessorThread.processStandardWebService(SOAPProcessorThread.java:231)
at com.glide.processors.soap.SOAPProcessorThread.processBody(SOAPProcessorThread.java:209)
at com.glide.processors.soap.SOAPProcessorThread.processRequest(SOAPProcessorThread.java:176)
at com.glide.processors.soap.SOAPProcessorThread.run0(SOAPProcessorThread.java:131)
at com.glide.util.ParentedThread.run(ParentedThread.java:51)
Caused by: ReferenceError: "cmdb" is not defined. (sys_script.154e085e1ba48d50c532da8ee54bcb1d.script; line 110)

Resolution

The customer's ServiceNow team to check the error message related to business rule &#39 and resolve the issue, once the ServiceNow issue is resolved it will return a successful error code and message to NIM and it will not recreate the incidents again.

Additional Information

How to enable debug between Spectrum and Service Desk (NIM) Integration
https://knowledge.broadcom.com/external/article?articleNumber=230119