Flat File masking gives java.nio.charset.MalformedInputException using FDM

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

When applying masking on a Flat file through FDM, we are getting the below error:
While masking I am getting the following error:

>2020-02-12 00:47:42.035 File column <COL_NAME> will be masked replacing digits with digits,  
>2020-02-12 00:47:42.035 lowercase letters with lowercase letters, and uppercase letters with uppercase letters 
>2020-02-12 00:47:42.035 The mask will create unique values assuming the original values are unique 
>2020-02-12 00:47:42.035 *************************** 
>2020-02-12 00:47:42.035  
>2020-02-12 00:47:42.035 starting masking at 2020.02.12 00:47:42.035 EST 
>2020-02-12 00:47:42.035 java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1
> at java.io.BufferedReader$1.hasNext(Unknown Source)
> at java.util.Iterator.forEachRemaining(Unknown Source)
> at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
> at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
> at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
> at java.util.stream.LongPipeline.reduce(Unknown Source)
> at java.util.stream.LongPipeline.sum(Unknown Source)
> at com.grid_tools.products.datamasker.Datamasker.getNumberOfLines(Datamasker.java:9336)
> at com.grid_tools.products.datamasker.Datamasker.doFileMasking(Datamasker.java:12048)
> at com.grid_tools.products.datamasker.Datamasker.main(Datamasker.java:4786)
>Caused by: java.nio.charset.MalformedInputException: Input length = 1
> at java.nio.charset.CoderResult.throwException(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)

Environment

Test Data Manager
TDM Portal
FDM
Fast Data Masker

Cause

After analysis, the input file which you have used is not in UTF-8 format data.
The input file is partially encoded in EBCDIC, which is not accepted by FDM.

Resolution

Use the proper format of UTF-8 input data file to mask in FDM.

To convert EBCDIC  to UTF-8 file, you can do the following

 

1. For single files, open the flat file in plain Notepad. Do a Save as and choose Encoding as UTF-8

 

NAME: Convert-Encoding.ps1