In this situation no changes are required to the data_engine settings. The NAS_TRANSACTION_LOG table is managed by the NIS BRIDGE settings. These tables are maintained by the nas using the NiS bridge configuration data. To keep the size of these tables in the NiS database manageable, they are automatically compressed at the configured administration interval, see the following documentation for more details:
The NiS Bridge Tab - https://docops.ca.com/ca-unified-infrastructure-management-probes/ga/en/alphabetical-probe-articles/nas-alarm-server/nas-im-configuration/the-nas-setup-tab#ThenasSetupTab-TheNiSBridgeTab
We've seen previously if the volume of data is large and the maintenance job cannot complete within the scheduled time frame, it will cancel the maintenance job and not delete any data. That could be the case here, if you manually delete the unwanted data from the NAS_TRANSACTION_LOG we would recommend taking a backup of the UIM database or table prior to removing the data (if possible) and stopping (deactivating) the data_engine and nas probes.