Flat File masking gives java.nio.charset.MalformedInputException using FDM

book

Article ID: 145017

calendar_today

Updated On:

Products

CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

When applying masking on Flat file through FDM, getting below error:

 

While masking I am getting the following error:

 

>2020-02-12 00:47:42.035 File column <COL_NAME> will be masked replacing digits with digits,  

>2020-02-12 00:47:42.035 lowercase letters with lowercase letters, and uppercase letters with uppercase letters 

>2020-02-12 00:47:42.035 The mask will create unique values assuming the original values are unique 

>2020-02-12 00:47:42.035 *************************** 

>2020-02-12 00:47:42.035  

>2020-02-12 00:47:42.035 starting masking at 2020.02.12 00:47:42.035 EST 

>2020-02-12 00:47:42.035 java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1

> at java.io.BufferedReader$1.hasNext(Unknown Source)

> at java.util.Iterator.forEachRemaining(Unknown Source)

> at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)

> at java.util.stream.AbstractPipeline.copyInto(Unknown Source)

> at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)

> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)

> at java.util.stream.AbstractPipeline.evaluate(Unknown Source)

> at java.util.stream.LongPipeline.reduce(Unknown Source)

> at java.util.stream.LongPipeline.sum(Unknown Source)

> at java.util.stream.ReferencePipeline.count(Unknown Source)

> at com.grid_tools.products.datamasker.Datamasker.getNumberOfLines(Datamasker.java:9336)

> at com.grid_tools.products.datamasker.Datamasker.doFileMasking(Datamasker.java:12048)

> at com.grid_tools.products.datamasker.Datamasker.main(Datamasker.java:4786)

>Caused by: java.nio.charset.MalformedInputException: Input length = 1

> at java.nio.charset.CoderResult.throwException(Unknown Source)

> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)

> at sun.nio.cs.StreamDecoder.read(Unknown Source)

> at java.io.InputStreamReader.read(Unknown Source)

> at java.io.BufferedReader.fill(Unknown Source)

> at java.io.BufferedReader.readLine(Unknown Source)

> at java.io.BufferedReader.readLine(Unknown Source)

> ... 13 more

 

Cause

After analysis, the input file which you have used is not in UTF-8 format data. The input file is partially encoded in EBCDIC, which is not accepted by FDM.
Please use proper format of UTF-8 input data file to mask.

Environment

Release : 4.6 and up

Component : CA Test Data Manager

Resolution

Use proper format of UTF-8 input data file to mask in FDM.

To convert EBCDIC  to UTF-8 file, you can do the following

 

1. For single files, open the flat file in plain Notepad. Do a Save as and choose Encoding as UTF-8

2. To Convert Files To UTF-8 Encoding (in Bulk), you can download this utility listed in the link below and pass in the source and destination directories. 

https://gallery.technet.microsoft.com/scriptcenter/Convert-Files-To-UTF-8-5c58db5e

NAME: Convert-Encoding.ps1 

 

Attachments