While using FDM 4.10.1014.0 to mask a large amount of Azure Synapse data, the masking job fails after roughly 25 hours with the following error:
Message: 112007;Exceeded the memory limit of 20 MB per session for prepared statements. Reduce the number or size of the prepared statements.
SQLSTATE: S0001
Error code: 112007
FDM 4.10.1014.0 or greater
The error that was thrown is from Azure Synapse. A Google search shows that by default, Azure Synapse has a limitation of 20MB of allocated memory per Synapse sessions for prepared statement.
Cause:
This is a known limitation for Synapse. Since an external table in Synapse by definition is mapped to a file, this can be any type, for example parquet#.
Broadcom's recommendation at this time is to create an external table mapped to a parquet file based on the source table.
Then mask the data contained within the parquet file using our the latest FDM release. Afterward's, update the source table from the masked external table.
The latest TDM and FDM patches are available for download from Test Data Manager (TDM) Support patches