Splunk importer jobs run long, timeout, and ingest duplicate records after upgrading the importer to version 6.5.4

Products

• Information Centric Analytics
• Data Loss Prevention Core Package

Issue/Introduction

After upgrading ICA to version 6.5.4, the duration of Splunk data source queries increases and may fail with a timeout error such as this:

[1:ERROR] SplunkSearchSession.WaitForCompletion() Inner Exception: Timeout waiting for Job slice to complete after 3600 seconds.

Jobs that complete without errors may ingest duplicate records into the staging table.

Environment

Release : 6.5.4

Component : Splunk Importer

Cause

Enhancements to data type checking in 6.5.4 resulted in datetime values being recognized and stored in ISO8601 format; however, the stored procedure spIW_RunSplunkImporter had not been updated to handle this format, which resulted in it returning a NULL value for datetime fields used as the watermark column in the IW data source query. Consequently, old and duplicate records were returned, increasing the quantity of data to be processed. The corresponding increase in processing time may exceed the timeout value previously specified, resulting in job failure.

Resolution

To update the importer and stored procedure to properly handle datetime watermark values, follow this procedure:

1. Download the updated Splunk Importer 6.5.4 binaries [attached to this KB article as SplunkImporter_654_20200925_01_1601329720980.zip]
2. Extract the contents of the downloaded archive
3. Create a backup of the contents of the existing Splunk importer folder
4. Copy and paste the contents of the extracted Splunk archive into the existing Splunk importer directory
5. Overwrite any files when prompted
6. Download the script to update spIW_RunSplunkImporter [attached to this KB article as spIW_RunSplunkImporter_1601329366032.sql]
7. Open SQL Server Management Studio (SSMS)
8. In the Connect to Server window, select the following:
   • Server type: Database Engine
   • Server name: RiskFabric database hostname
   • Authentication: account credentials to alter the RiskFabric database
   • Click Connect
9. Open the File menu and select Open > File...
10. Locate and select the script downloaded from this article in step 6 and click Open
11. Select the RiskFabric database in the drop-down list of databases in the Standard toolbar (RiskFabric is the default name of the ICA database but may differ in your environment)

    

12. Open the Query menu and select Execute, or press the F5 function key to execute the script