java.lang.Exception: Error: Invalid bulk event file -- no manifest

Products

• Data Center Security Server Advanced

Issue/Introduction

When attempting to re-process, into the SCSPDB database, a group of  event files (*.csv) which are compressed in one or many *.zip files-- fails to process. It terminates with an error similar to the following occurred:

SCSP Bulk Event Loader

File Path/Name:      C:\Temp\Events\SISRTEvents28197.zip
Integrity Override:  ON
Destination Table:   CSPEVENT
Database Connection: DEFAULT
Database User:       DEFAULT

Processing events file...
Error loading events:
java.lang.Exception: Error: Invalid bulk event file -- no manifest.
Please be sure the zipfile is an event file.
        at com.symantec.sis.bulkeventloader.FileLoader.process(FileLoader.java:169)
        at com.symantec.sis.bulkeventloader.BulkLoader.processEventFile(BulkLoader.java:505)
        at com.symantec.sis.bulkeventloader.BulkLoader.main(BulkLoader.java:91)
No events processed.

 

Environment

Data Center Security 6.9.1

 

 

Cause

In order for the data loading process to function correctly when processing events en masse--it needs to know the Guid of the computer from which the events came, and match it with a valid asset within the database.  In order for it to do this it needs to be provided with the correct Guid within a file known as the "manifest".

1. The manifest file is made up of a text file where the name does not matter, but the extension must be .mnf    Example:  Events.mnf
2. The manifest file has one line. Guid:<computerGuid>      Example:   Guid:ABCDEFGHIJKLMNOPQRSTUVWXYZ-Abcdefg+hijkLMNop:-qrstuvwx:yz012345678:-9098
3. The manifest file must be added to each zip file containing events needing to be processed again.

Resolution

Steps:

1. Open one of the .csv files compressed in the zip file

2. On the very first line the Guid will be the long series of characters between the 14th and 15th commas. 

Example:  MSOF,0,2015-03-23 06:16:14.000 Z+0530,I,,,,,,,<hostname>,unknown,unknown,windows,ABCDEFGHIJKLMNOPQRSTUVWXYZ-Abcdefg+hijkLMNop:-qrstuvwx:yz012345678:-9098,C:\Program Files (x86)\Symantec\Data Center Security Server\Agent\sdcsslog\SISIDSEvents.csv,,E,6.5.0.355

3. Create a text file:  Events.mnf

4. Open the newly created file and insert the line:  Guid:

5. Next to the "Guid:" heading paste the Guid extracted from 14th position in the .csv file.  Example:   Guid:ABCDEFGHIJKLMNOPQRSTUVWXYZ-Abcdefg+hijkLMNop:-qrstuvwx:yz012345678:-9098

6. Save the manifest file and insert it into each zip file containing event files needing to be processed.   The zip file should contain events from one specific computer, not a mix of computer agents.

NOTE: If you are unsure if the Guid being used is correct or not one of the following SQL queries can be run via SQL Management Studio against SCSPDB

-- Run copy the Guid from the first line of a .csv event file and paste it into the value for the variable @Guid

declare @Guid nvarchar(255) = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ-Abcdefg+hijkLMNop:-qrstuvwx:yz012345678:-9098'

select  Hostname, IPAddress, OSType, OSVersion, AgentAge, LastAppRequestDate
from ASSET_VW
where Guid = @Guid

 

If the above query returns nothing or if it returns the name of a computer not expected you can look up the current operating Guid for the computer by plugging the computer's name into the following query and running it against the database:

Replace 'Machine1' with the computer name in the following example:

declare @computerName nvarchar(50) = 'Machine1'

select  Hostname, IPAddress, OSType, OSVersion, AgentAge, LastAppRequestDate, Guid
from ASSET_VW
where Hostname = @computerName