An approach to mask multiple flat files with variable record details

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

We have requirements for masking several data files that are all fixed format files with:

• 1 Header row and
• 1 Trailer Row
•  15 record Types that contain 1 or more fields that need to be masked
• 4 Record Types that contain no field/s to be masked

We have over 5000 data files to be masked. These data files contain several record types with the possibility of having different PII/PHI elements in a different location depending upon the Record Type field, which is the 6th position (3 chr). 

Kindly provide guidance on how to go about masking such files. 

As an extension to our solution, kindly suggest a way to "automate" the process. So, when a file is dropped into the folder, it would trigger the FDM, with reference to a common _DM file and after processing has completed, the original and scramble files are moved to another location. 

Environment

Release : 4.10

Resolution

Here is an example of masking a file with multiple record types.

In the file, Emp_data.dat, there are 3 record types.

employee

employee address

employee creditcard

 

You need to create a .definition file (dm.txt file) for each record type.  In each .dm.txt file, you define how FDM should determine which record is which?

For example, in the employee.dm.txt file you see

 

DELIM=FIXED,RECTYPE=RECORD_ID:"EMP"
RECORD_ID,3
EMPLOYEE_ID,4
FIRST_NAME,20
LAST_NAME,20
EMAIL,20
PHONE_NUMBER,15
BIRTH_DATE,10,"YYYY-MM-DD"
EMP_SSN,10
EMP_PASSWORD,20
HIRE_DATE,10,"YYYY-MM-DD"
SALARY,5
EXPENSES,5
SAVINGS,5
EMP_RECORD_ID,8

 

What is in bold drives this.

 

It doesn't have to be the first column.

 

Once this is all done, you have to create or update the connection file .txt file used by FDM, to include the multiple .dm.txt files. The easiest way to do this is to create the connection profile in FDM first, by selecting a single dm.txt file, and save the connection, and close the FDM UI. 

 

Then Windows File Explorer, navigate to %AppData%\Grid-Tools\FastDatamasker, and locate the connection file you created. This is the default location where FDM saves the connection files. It's possible this has been changed in your FDM configuration, so you may need to find out where your connection files are saved. In the texted editor of your choice, open the connection file, and modify the file by adding in all the dm.txt files, separated by a comma. 

For example:

datasource=file
file=C:/MASK/MyTestDirectory/MyTestDataFiles/MyTest_U.DAT.txt
filedef=MyTest_RECTYPTE_B_dm.TXT,MyTest_RECTYPTE_BNP_DM.TXT,MyTest_RECTYPTE_BNP_DM.TXT,MyTest_RECTYPTE_IC_DM.TXT,MyTest_RECTYPTE_L_DM.TXT,MyTest_RECTYPTE_LNP_DM.TXT,MyTest_RECTYPTE_LNP_DM.TXT,MyTest_RECTYPTE_PLN_DM.TXT,MyTest_RECTYPTE_PLN2_DM.TXT,MyTest_RECTYPTE_PLS_DM.TXT,MyTest_RECTYPTE_PO_DM.TXT,MyTest_RECTYPTE_PRP_DM.TXT,MyTest_RECTYPTE_SFL_DM.TXT,MyTest_RECTYPTE_SPF_DM.TXT,MyTest_RECTYPTE_T_DM.TXT
filedefdir=C:/MASK/MyTestDirectory/MyTestDefinitionFiles
CSPM_CLIENT_CONFIG_XML=C:\\cspm\\cloakware\\cspmclient\\config\\cspm_client_config.xml
JAVA_LIBRARY_PATH=C:\\cspm\\cloakware\\cspmclient\\lib;c:\\cspm\\cloakware\\cspmclient\\jre\\bin;.\\SQLSERVER_DLLs\\x64 

 

Where:

• • file= is the complete path to the data file we are masking
  • filedef= is the list of definition files, separated by a comma, that you want to load.
  • filedefdir= is the path to the directory containing the definition files.

 

 

When FDM connects, we see tabs open for each of the DM files that were loaded. FDM will treat each .dm.txt file as if it were a table.

 

 

Note: you do not need to include the Header, and Trailer. All that is required is the DELIM and RECTYPE. Also, FDM doesn't handle spaces well, so if you have a space (for example, First Name) you will want to delete the space or replace it with an underscore. 

 

For automating the process, once the FDM configuration is in place, you can design a Javelin Workflow to check the file location where the .dm.txt files will be placed, and kickoff a masking job if any files have been added to the directory. Once masking completes, as a post process, Javelin can move/archive the processed files to another directory.

Additional Information

Attached to this KB article is a set of sample files in case you need them.