JsonImporter fails during staging of E-mail and Web Security.cloud events

Products

• Information Centric Analytics

Issue/Introduction

The following errors appear in the JSONImporter log when importing events from E-mail and Web Security.cloud:

2021-02-21 12:20:59,402 JsonImporter.exe [1:INFO] ApiClient.GetResponse() Streaming response to temp file C:\DatabaseUtilities\JsonImporter\Malware_20210221122059_p44vr2vb.eq2.json
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] ApiClient.LogError() Could not retrieve Malware (attempt 0 out of 4) saved response file 'C:\DatabaseUtilities\JsonImporter\Malware_20210221122059_p44vr2vb.eq2.json'
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] ApiClient.LogError() exception: Value cannot be null.
Parameter name: source
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] ApiClient.LogError() System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)
   at JsonImporterUtils.SymantecEmailApi.SqlPersistor.Stage(Endpoint endpoint, IEnumerable`1 data)
   at JsonImporterUtils.SymantecEmailApi.SymantecEmailImporter.GenericParseAndStage(IPersistor persistor, String s, Endpoint endpoint)
   at JsonImporterUtils.ApiClient.GetResponse(Endpoint endpoint, IRequest request, Func`4 fParseAction)
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] ApiClient.LogError() Http status code OK
2021-02-21 12:20:59,496 JsonImporter.exe [1:WARN] ApiClient.GetResponse() Checking for retry
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] ApiClient.GetResponse() Encountered error during download. Check ApiEvents table, log file and C:\DatabaseUtilities\JsonImporter\Malware_20210221122059_p44vr2vb.eq2.json for additional details.
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] SymantecEmailImporter.Import() Exception on endpoint Malware :
2021-02-21 12:20:59,496 JsonImporter.exe [1:ERROR] SymantecEmailImporter.Import() System.Exception: Error occurred while retrieving data from the Api
   at JsonImporterUtils.SymantecEmailApi.SymantecEmailImporter.Import()

Environment

Release : 6.5.x

Component : JsonImporter

Cause

The JsonImporter does not handle NULL values. This is a known issue that will be addressed in the next release of ICA.

Resolution

The JSONImporter should be configured to query the Malware feed no less frequently than once per hour. Because of the way in which the API retrieves data in response to a query, larger query intervals will cause the request to time-out.