Splunk data source query create job failure

Products

• Information Centric Analytics

Issue/Introduction

When running a Splunk data source query, the data source query job fails and the Splunk importer log captures the following errors:

[4:ERROR] SplunkApi.MoveNext() SplunkApiNonTimeSeries Create job failed.
[4:DEBUG] SplunkApi.MoveNext() SplunkApiNonTimeSeries IsCancellationRequested: True
[4:ERROR] SplunkApi.MoveNext() SplunkApiNonTimeSeries Create job failed.
[1:WARN] QueryRunnerBase.LoadFromSplunk() ProcessTimeSlice success=false
[1:ERROR] QueryRunnerBase.OnException() OnException()
System.Exception: *Failure during LoadFromSplunk*
   at BayDynamics.Splunk.Utils.Search.QueryRunnerBase.Execute() in <path>\QueryRunnerBase.cs:line 162
[1:ERROR] Program.Main() System.Exception: Failure during LoadFromSplunk
   at BayDynamics.Splunk.Utils.Search.QueryRunnerBase.Execute() in <path>\QueryRunnerBase.cs:line 180
   at BayDynamics.Splunk.Program.Main(String[] args) in <path>\Program.cs:line 156
...
=========================================================================================================
EXIT SPLUNK IMPORTER RUN ID: dee2ac9c-2acd-4db6-9598-a05139b87a0e
EXIT STATUS: Error Loading from Splunk
=========================================================================================================

The time elapsed between the time of job creation and cancellation is greater than or equal to 30 seconds (note the values in red):

2020-09-30 18:15:01,100 [1:DEBUG] SplunkApi.MoveNext() Creating job...
2020-09-30 18:15:01,147 [1:DEBUG] SplunkApi.MoveNext() Awaiting Job Creation...
2020-09-30 18:15:31,219 [10:ERROR] SplunkApi.MoveNext() TaskCanceledException Count 0 - continue.

Environment

Release : 6.5.4

Component : Splunk Import Utility

Cause

The Splunk SDK contains a hardcoded timeout of 30 seconds (30000 ms) to pass query configuration parameters to the Splunk server during the Splunk job creation process. This timeout in the SDK overrides any of the timeout settings passed by the ICA Splunk import utility.

Resolution

This bug in the SDK has been reported to Splunk. Until a fix to the SDK is released, reducing the size and complexity of Splunk queries has been found to be an effective workaround.