File Masking Questions

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

Some queries related to file masking:

1. Local File Masking : This is not centralized, meaning, place the input file on a local C:\User_1\Input and scripts are created based on the User_1 path details .
   Problem : If user_2 has to run the masking, will have to make changes to path details in the Scripts ()same goes with User_3/User_4 etc .
   What will be the best way to tackle this?
2. Does Portal support File masking -> Not performing Audit and Data Model for files, Just wondering if FDM is the only option to perform 'File masking'
3. What are the File Connector Options for files : SFTP, FTP, Hadoop?
   Portal should be able to take the server address/ip address, Port, path etc. As of now, when a File is selected, it allows to enter the local File path details. 
4. Any documentation on Parquet file masking.

 

 

 

Environment

Release : 4.10

Resolution

Answers:

Questions 1:  Customers have different approaches to this, and it determines what is best for the use case. The scripts generated by FDM are primarily used as templates for running the masking job on a remote system. Many customers modify these scripts to include bash/shell variables to make them more dynamic for orchestration/automation purposes.

However, if just looking for centralized storage, could configure FDM to use a default/shared location, so all users could easily find the scripts. Although, it's not recommended to use network shared drives, for the FDM job logs, as this can impact the performance of the masking job. To configure a default drive location in FDM, from the FDM UI, click the Settings Menu pick, and click on "Set Default Directories".

If wanting to run the scripts from a centralized location, suggest installing FDM on a shared/centralized server, setting the default file locations to a local drive, and all users log into the shared server to run FDM, to execute the masking jobs, or run the scripts.

For FDM Best Practices see https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-10/provisioning-test-data/mask-production-data-with-fast-data-masker/fast-data-masker-best-practices.html

Question 2:  The short answer to this is yes. For any file type that FDM supports for data masking, TDM Portal also supports it. TDM Portal uses the FDM masking engine for all the masking jobs. If wanting to use Portal, the file must be a registered object within the Data model. Since not doing Data modeling, FDM is your best option.

TDM 4.10 Supported Non-Relational Data Sources:

Data Source	Dynamic Test Data Reservation - TDM Portal	Data Generation - TDM Portal	Data Generation - Datamaker	Data Masking	Data Subset	Test Match	vTDM	Data Modeling and PII Audit
SQL Files	Not Supported	Certified	Certified	Not Supported	Not Supported	Not Supported	Not Supported	Not Supported
CSV Files	Not Supported	Certified	Certified	Certified	Certified	Not Supported	Not Supported	Not Supported
Fixed Definition Files	Not Supported	Certified	Certified	Certified	Certified	Not Supported	Not Supported	Not Supported
XML Files	Not Supported	Certified	Certified	Certified	Not Supported	Not Supported	Not Supported	Not Supported
Excel Files	Not Supported	Supported	Supported	Supported	Supported	Not Supported	Not Supported	Not Supported
HTML Files	Not Supported	Not Supported	Supported	Not Supported	Not Supported	Not Supported	Not Supported	Not Supported
TXT Files	Not Supported	Supported	Supported	Supported	Supported	Not Supported	Not Supported	Not Supported
VSAM/ISAM	Not Supported	Not Supported	Supported	Supported	Supported	Supported	Not Supported	Not Supported
JSON Files	Not Supported	Supported	Supported	Supported	Not Supported	Not Supported	Not Supported	Not Supported
Hadoop	Not Supported	Not Supported	Not Supported	Certified	Not Supported	Not Supported	Not Supported	Not Supported

 

Questions 3: Correct,  FDM and TDM Portal typically use locally stored files. This is mainly for masking performance, however, should also be able to use a mapped network share or shared network drive, but there could be a slower performance due to the network connection. Neither FDM nor TDM Portal uses FTP/SFTP as a means to access files for masking. See https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-10/provisioning-test-data/mask-production-data-with-fast-data-masker/mask-stored-data/mask-data-stored-in-flat-files.html for more information.

As for Hadoop, FDM and Portal use FDM masking functions as Hive user-defined functions (UDFs) to mask data stored in Hadoop. The stored data must be structured data and must have a defined schema. The TDM installation media provides a MaskingSDK-4.10.01.zip, which contains JAR files that include Hive UDFs, which are developed based on a standalone Java masking library. The Java masking library includes Fast Data Masker masking functions. When executing these Hive UDFs (provided in the JAR file) in the Hadoop environment, performs the defined masking operations and mask the structured data.  For more information, see https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-10/provisioning-test-data/mask-production-data-with-fast-data-masker/mask-stored-data/mask-data-stored-in-hadoop.html.

Question 4:  The Parquet file is not a supported data source at this time. Submit an enhancement request to the TDM Product Management team, asking to consider this for a future enhancement.