Indexing EDMI fails on large files

Products

• Data Loss Prevention Core Package

Issue/Introduction

Due to the large volume of data I had to divide my EMDI source file into 4 separate files when divided the source file into 4 separate txt files each of the 4 files indexes successfully individually.  When I combine all 4 .txt files into one huge .txt file and copy it to Enforce the indexing job fails.  

 

Environment

Release : 15.8 MP3

Cause

In the EDM indexer logs the error was observed 

"Message: Exceed max number of duplicate cells"

come to find out that when the user was merging the documents, the user was leaving the headers, which represented duplication in the source data.

Resolution

Advised that the source data still had too many duplicate cells, and would need to be cleaned up before indexing, that the user would need to remove the extra headers when merging the documents.