Splunk data source query job staging failure

Products

• Information Centric Analytics

Issue/Introduction

Step 5 of the data source query job RiskFabric_IW_DataSourceQueryID_n is failing. The data source is Splunk and the Splunk importer logs capture the following:

2023-01-19 21:17:50,255 [1:DEBUG] QueryRunnerBase.ProcessSlice() ProcessTimeSlice(0) =======================================
2023-01-19 21:17:50,261 [1:DEBUG] SplunkApi.MoveNext() Creating Non Time Series Job...
2023-01-19 21:17:50,357 [1:DEBUG] SplunkApi.MoveNext() Awaiting Non Time Series Job Creation...
2023-01-19 21:17:50,429 [4:ERROR] SplunkApi.MoveNext() SplunkApiNonTimeSeries Create job failed.
2023-01-19 21:17:50,430 [4:DEBUG] SplunkApi.MoveNext() SplunkApiNonTimeSeries IsCancellationRequested: True
2023-01-19 21:17:50,430 [4:ERROR] SplunkApi.MoveNext() SplunkApiNonTimeSeries Create job failed.
2023-01-19 21:17:50,431 [1:INFO] SplunkSearchSession.Start() CreateJobAsync TimeSpan 0s
2023-01-19 21:17:50,431 [1:WARN] QueryRunnerBase.LoadFromSplunk() ProcessTimeSlice success=false
2023-01-19 21:17:50,433 [1:ERROR] QueryRunnerBase.Execute() OnException()
System.Exception: Failure during LoadFromSplunk
   at BayDynamics.Splunk.Utils.Search.QueryRunnerBase.Execute()
2023-01-19 21:17:50,434 [1:ERROR] Program.Main() System.Exception: Failure during LoadFromSplunk
   at BayDynamics.Splunk.Utils.Search.QueryRunnerBase.Execute()
   at BayDynamics.Splunk.Program.Main(String[] args)
2023-01-19 21:17:50,458 [1:INFO] Program.AttemptWatermarkUpdate() 
=========================================================================================================
Watermark Update: -1 Records Updated
2023-01-19 21:17:50,458 [1:INFO] Program.Exit() 
=========================================================================================================
EXIT SPLUNK IMPORTER RUN ID: 39cecee4-3f47-479b-b519-501135130e9f
EXIT STATUS: Error Loading from Splunk
=========================================================================================================

Environment

Release : 6.x

Cause

The pertinent reference in the Splunk importer log is the following:

2023-01-19 21:17:50,261 [1:DEBUG] SplunkApi.MoveNext() Creating Non Time Series Job...
2023-01-19 21:17:50,357 [1:DEBUG] SplunkApi.MoveNext() Awaiting Non Time Series Job Creation...
2023-01-19 21:17:50,429 [4:ERROR] SplunkApi.MoveNext() SplunkApiNonTimeSeries Create job failed.
2023-01-19 21:17:50,430 [4:DEBUG] SplunkApi.MoveNext() SplunkApiNonTimeSeries IsCancellationRequested: True
2023-01-19 21:17:50,430 [4:ERROR] SplunkApi.MoveNext() SplunkApiNonTimeSeries Create job failed.

This message indicates the Splunk search head was unable to create a job to process the query. This can be caused by an unresponsive search peer, or the account under which the query is being run has exceeded its query quota.

Resolution

Contact your Splunk administrator for assistance with determining why Splunk is unable to create a non-time series job.