When SMSMSE is enabled and set to scan SMTP messages; Committed Bytes in use %, Private Bytes, Virtual Bytes and Handle Count of the Edgetransport.exe process continue to increase until the Exchange Server becomes unstable and the server either hangs or the EdgeTransportService restarts.
Handles allocated by SMSMSE transport agents during SMTP scanning are not always freed as expected.
This issue is fixed in SMSMSE 7.5.3. Upgrade to 7.5.3 to fix this problem.
Workaround
There are 3 potential workarounds for this issue:
1. A hotfix has been created for version 7.5.1 of SMSMSE, attached to this article as HF_3800419_3.rar.
2. Disable in memory scanning and conduct transport scans on disk by creating the following registry key and assigning it the value 0:
3. Periodically restart the Exchange Transport service to free up the memory consumed by this leak.