Flat File masking gives java.nio.charset.MalformedInputException using FDM
search cancel

Flat File masking gives java.nio.charset.MalformedInputException using FDM


Article ID: 145017


Updated On:


CA Test Data Manager (Data Finder / Grid Tools)


When applying masking on a Flat file through FDM, we are getting the below error:
While masking I am getting the following error:

>2020-02-12 00:47:42.035 File column <COL_NAME> will be masked replacing digits with digits,  
>2020-02-12 00:47:42.035 lowercase letters with lowercase letters, and uppercase letters with uppercase letters 
>2020-02-12 00:47:42.035 The mask will create unique values assuming the original values are unique 
>2020-02-12 00:47:42.035 *************************** 
>2020-02-12 00:47:42.035  
>2020-02-12 00:47:42.035 starting masking at 2020.02.12 00:47:42.035 EST 
>2020-02-12 00:47:42.035 java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1
> at java.io.BufferedReader$1.hasNext(Unknown Source)
> at java.util.Iterator.forEachRemaining(Unknown Source)
> at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
> at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
> at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
> at java.util.stream.LongPipeline.reduce(Unknown Source)
> at java.util.stream.LongPipeline.sum(Unknown Source)
> at com.grid_tools.products.datamasker.Datamasker.getNumberOfLines(Datamasker.java:9336)
> at com.grid_tools.products.datamasker.Datamasker.doFileMasking(Datamasker.java:12048)
> at com.grid_tools.products.datamasker.Datamasker.main(Datamasker.java:4786)
>Caused by: java.nio.charset.MalformedInputException: Input length = 1
> at java.nio.charset.CoderResult.throwException(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)


Test Data Manager
TDM Portal
Fast Data Masker


After analysis, the input file which you have used is not in UTF-8 format data.
The input file is partially encoded in EBCDIC, which is not accepted by FDM.


Use the proper format of UTF-8 input data file to mask in FDM.

To convert EBCDIC  to UTF-8 file, you can do the following


1. For single files, open the flat file in plain Notepad. Do a Save as and choose Encoding as UTF-8

2. To Convert Files To UTF-8 Encoding (in Bulk), you can download this utility listed in the link below and pass in the source and destination directories. 


NAME: Convert-Encoding.ps1