Set up the generator - for a table -for a simple PoC involving Source table to Target table new data generation with a few PII/PHI/PFI info generated by function.
Source table has 4.59 million rows and it fails to generate new data. (after changing the Properties file per suggestion from support.)
The job failed to complete (tried more than a few times).
The bug seems to be that when the source table is large, it is trying to read all rows before generating, updating and committing them in the chunks of 500K.
To test the hypothesis, changed the "Read SQL to be limited to only 5000 rows and the job ran quickly within 10 minutes.
All supported versions of TDM.
10.235 (version). DB - snowflake
Product limitation.
Doing the generation in chunks is the way you should be doing it. It will not work doing it with all those millions of rows.