Bug: Generator in Snowflake fails to complete/save any rows
search cancel

Bug: Generator in Snowflake fails to complete/save any rows

book

Article ID: 367093

calendar_today

Updated On:

Products

CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

Set up the generator - for a table -for a simple PoC involving Source table to Target table new data generation with a few PII/PHI/PFI info generated by function. 

Source table has 4.59 million  rows and it fails to generate new data. (after changing the Properties file per suggestion from support.)

The job failed to complete (tried more than a few times). 

The bug seems to be that when the source table is large, it is trying to read all rows before generating, updating and committing them in the chunks of 500K. 

To test the hypothesis, changed the "Read SQL to be limited to only 5000 rows and the job ran quickly within 10 minutes. 

 

Environment

All supported versions of TDM.

10.235 (version). DB - snowflake

Cause

Product limitation.

Resolution

Doing the generation in chunks is the way you should be doing it. It will not work doing it with all those millions of rows.