When running Symantec Control Compliance Suite "collection - evaluation - reporting" jobs, generating large amounts of evidence data (over 1Gb of data per job) - the Data Processing Service (DPS) process could run out of memory and throw an "System.OutOfMemoryException" error.
2011-10-20 00:27:28.248,2011-10-20 01:27:28.248,DPS_HOSTNAME,Error,ETLBusiness,Blade.WorkerProcess,1752,Dispatcher Submit Queue:9,9,ProcessQuery,,0,0,"Query:
,'NONE' AS AssetVersion
,@JobID AS evaluationJobID
,[Outcome] AS checkResultDisplayValue
,[RawOutcome] AS checkResultValue
,[Exempt] AS checkApplyException
,[RiskScore] AS checkRiskScore
SELECT TargetID, AssetID
WHERE JobID = @JobID
ON [R_CheckResults].[TargetID] = Targets.TargetID"
2011-10-20 00:27:28.248,2011-10-20 01:27:28.248,DPS_HOSTNAME,Error,ETLBusiness,Blade.WorkerProcess,1752,Dispatcher Submit Queue:9,9,ProcessQuery,,0,0,Time taken for processing query : 00:00:23.6712690
2011-10-20 00:27:28.248,2011-10-20 01:27:28.248,DPS_HOSTNAME,Error,ETLBusiness,Blade.WorkerProcess,1752,Dispatcher Submit Queue:9,9,ProcessQuery,,0,0,"System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Symantec.CSM.ETL.Extractors.Database.DatabaseExtractor.ProcessQuery(Query query, List`1 extendedDataExtractors)
at Symantec.CSM.ETL.Extractors.Database.DatabaseExtractor.ProcessQueryGroup(QueryGroup queryGroup)
at Symantec.CSM.ETL.Extractors.Database.DatabaseExtractor.Process(DataExtractionParameterCollection dataExtractionParams)
at Symantec.CSM.ETL.Extractors.StandardEvaluations.AutoSyncStandardsEvaluationExtractor.Process(DataExtractionParameterCollection dataExtractionParams)
at Symantec.CSM.ETL.Processor.ETLProcessor.Process(ExtractorType typeofETLProcessToFork, DataExtractionParameterCollection dataExtractionParams, Guid loadID)"
2011-10-20 00:27:28.248,2011-10-20 01:27:28.248,DPS_HOSTNAME,Error,ETLBusiness,Blade.WorkerProcess,1752,Dispatcher Submit Queue:9,9,Process,,0,0,Exception happed in AutoSyncStandardEvaluation extractor
The Data Processing Service (DPS) is unable to cope with the large amount of data offered during the synchronization process.
To avoid this issue you could run the job on less assets at the time and thus reducing the amount of data to be processed during synchonization. Or reduce the amount of data to be collected per asset. For example the Enterprise Security Manager "filefind" module could generate overwhelming amounts of evidence data. The suggestion here is to reduce the amount of data collected by limiting the amount of ESM messages collected by the ESM policy run.(maximum reported messages check).
To fix the issue permanently and without limiting the amount of evidence collected, the DPS code has been changed to better cope with large amounts of data. The fix is available in Control Compliance Suite PCU 2012-1 onwards.
To implement the fix two things need to be in place:
1) new code (install PCU 2012-1 http://www.symantec.com/docs/TECH181183 or later)
2) a configuration setting – (it's in the release notes on PCU2012-1)
You need to add/verify the EvidenceExtractionBatchSize setting in Blade.WorkerProcess.exe.config file of all the reporting DPS’s
This is the recommended configuration of that section.
<add key="CslaAuthentication" value="Windows" />
<add key="CslaDataPortalProxy" value="Symantec.CSM.ETL.Extractors.Policy.PMCSLAWcfProxy, Symantec.CSM.ETL.Extractors.Policy" />
<add key="RecordsBatchSize" value="35000" />
<add key="ThirdPartyEvidenceChunkSize" value="6000" />
<add key="EvidenceExtractionBatchSize" value="100" />
After that restart the DPS service.