Snowflake masking Performance is slow
Article ID: 271395
Updated On:
Products
Issue/Introduction
I am trying to mask a big table in Snowflake, about 200,000K rows, and it does not work . It just stops at the attached log step, which appears to be more of a performance issue. The same FDM script runs fine for a smaller number of records, in the 100's of records.
I have 300 million records in this table and most of our customer data is on Snowflake, and need a way to improve the masking performance.
Environment
Release : TDM Portal 4.10 and FDM 4.10
Cause
The slowness is related to the way Snowflake handles "update" requests. Snowflake is known to not handle updates to tables efficiently, which is the cause of the slowness. When performing an update to a Snowflake table, Snowflake updates the table 1 row at a time. Because FDM and Portal are designed to perform updates to tables, we will not see good performance, especially for large amounts of data.
Resolution
A better option is to use GT Datamaker, which does "In flight" masking, meaning instead of issuing updates to an existing table, Datamaker pulls the data from the source table, loads it into memory or a temporary table, masks the data, and inserts the masked data into a target table that can be used by testers.
To set up the ODBC connection in Datamaker for Snowflake, you need to install the Snowflake 32-bit driver, Datamaker is still a 32-bit application, and only uses the 32-bit drivers.
Feedback
