Splunk integrations and _time

Products

• Information Centric Analytics

Issue/Introduction

Splunk import jobs intermittently fail during processing with the following errors:

[4:ERROR] SplunkApi.MoveNext() TaskCanceledException Count 0 - continue.
[4:ERROR] SplunkApi.MoveNext() System.Threading.Tasks.TaskCanceledException: A task was canceled.
[4:ERROR] SplunkApi.MoveNext() Create job failed.
[1:ERROR] QueryRunnerBase.Execute() OnException()
[1:ERROR] Program.Main() System.Exception: Failure during LoadFromSplunk
EXIT STATUS: Error Loading from Splunk

Environment

Release : 6.x

Component : Splunk Importer

Cause

The Information Centric Analytics (ICA) Splunk importer relies on Splunk's internal _time field for both retrieving time series information and for storing the watermark value for subsequent pulls of data.

Splunk uses the field _time internally for timeboxing each search. If a field other than _time is used as the watermark reference value in ICA, there is a possibility that query results will duplicate data because of the difference between the reference value in ICA's watermark field and the values queried by Splunk against _time.

Resolution

Use _time as the reference time field for the watermark in your Splunk integration.