Splunk data source query connection closed abruptly

Products

• Information Centric Analytics
• Data Loss Prevention Core Package

Issue/Introduction

Errors are seen in the SQL Server Agent job history of one or more RiskFabric_IW_DataSourceQueryID_n jobs used to stage data from a Splunk data source:

Integration Wizard - DataSourceQueryID = n, Step  4 - Setup Logging for Staging Query

Where n is the ID of the affected data source query.

These errors are also recorded in the RiskFabric Log_DataTransformation table in the LogName field, with a reference to the SplunkImporter.exe executable in the LogDescription field.

Additional errors can be found in the Splunk log:

[1:DEBUG] QueryRunnerBase.ProcessSlice() ProcessTimeSlice(1) =======================================
[1:DEBUG] SplunkApi.MoveNext() Creating job...
[1:DEBUG] SplunkApi.MoveNext() Awaiting Job Creation...
[9:ERROR] SplunkApi.MoveNext() Create job failed. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

Note that 'ProcessTimeSlice(1)' is incremented and the error may occur during any time slice of the overall job.

Environment

Release : 6.5.x

Component : Splunk API Data Source Integration

Cause

The Splunk server has abruptly closed the connection created by the data source query job prior to query completion.

There can be any of a number of environmental causes for this, including firewall policies, network QoS, and depleted system resources on the Splunk server.

Resolution

The resolution of this issue will be dependent upon the underlying cause. Broadcom recommends engaging with Splunk and network administrators to assist with investigation and remediation.