Splunk importer jobs run long, timeout, and ingest duplicate records after upgrading the importer to version 6.5.4
search cancel

Splunk importer jobs run long, timeout, and ingest duplicate records after upgrading the importer to version 6.5.4


Article ID: 200431


Updated On:


Information Centric Analytics Data Loss Prevention Core Package


After upgrading ICA to version 6.5.4, the duration of Splunk data source queries increases and may fail with a timeout error such as this:

[1:ERROR] SplunkSearchSession.WaitForCompletion() Inner Exception: Timeout waiting for Job slice to complete after 3600 seconds.

Jobs that complete without errors may ingest duplicate records into the staging table.


Release : 6.5.4

Component : Splunk Importer


Enhancements to data type checking in 6.5.4 resulted in datetime values being recognized and stored in ISO8601 format; however, the stored procedure spIW_RunSplunkImporter had not been updated to handle this format, which resulted in it returning a NULL value for datetime fields used as the watermark column in the IW data source query. Consequently, old and duplicate records were returned, increasing the quantity of data to be processed. The corresponding increase in processing time may exceed the timeout value previously specified, resulting in job failure.


To update the importer and stored procedure to properly handle datetime watermark values, follow this procedure:

  1. Download the updated Splunk Importer 6.5.4 binaries [attached to this KB article as SplunkImporter_654_20200925_01_1601329720980.zip]
  2. Extract the contents of the downloaded archive
  3. Create a backup of the contents of the existing Splunk importer folder
  4. Copy and paste the contents of the extracted Splunk archive into the existing Splunk importer directory
  5. Overwrite any files when prompted
  6. Download the script to update spIW_RunSplunkImporter [attached to this KB article as spIW_RunSplunkImporter_1601329366032.sql]
  7. Open SQL Server Management Studio (SSMS)
  8. In the Connect to Server window, select the following:
    • Server type: Database Engine
    • Server name: RiskFabric database hostname
    • Authentication: account credentials to alter the RiskFabric database
    • Click Connect
  9. Open the File menu and select Open > File...
  10. Locate and select the script downloaded from this article in step 6 and click Open
  11. Select the RiskFabric database in the drop-down list of databases in the Standard toolbar (RiskFabric is the default name of the ICA database but may differ in your environment)

  12. Open the Query menu and select Execute, or press the F5 function key to execute the script


SplunkImporter_654_20200925_01_1601329720980.zip get_app
spIW_RunSplunkImporter_1601329366032.sql get_app