Setting up TLS encryption for SMTP Prevent


Article ID: 159402


Updated On:


Data Loss Prevention Network Prevent for Email


How to setup TLS encryption and what logs to set if further troubleshooting is required


Make sure that the following settings are set for proper logging on the SMTP Prevent.
Set the log levels on the Network Prevent for Email detection server
Within the ..\Protect\config directory adjust the file and set or assign the following value

com.vontu.mta.log.SmtpOperationalLogHandler.level = FINEST
java.util.logging.FileHandler.level = FINEST
com.vontu.mta.rp.level = FINEST

Then restart the detection server service

(1) Please verify that the wording of "STARTTLS" has presented in the "RequestProcessor.AllowExtensions" whitelist.

If not, add it in, and recycle the SMTP prevent server and try it again.
To edit, log into the DLP Console as Administrator and go to
System > Servers > Overview > Click on the SMTP Prevent Server > Click on 'Server Settings'

(2) If above still not working, then the problem may relate to the format of the certificate that imported from MTA.

Proper X509 certificates need to be installed on prevent in order for this to work securely.
(In this POA - ie. trustedcert.pem) .pem file -- Privacy Enhanced Mail Certificate

Follow steps below for TLS setup

A. Down Stream MTA

1. Generate a new public/private key pair:

openssl req -new -key ca.key -out ca.csr

2. Generate a .pem file by signing a certificate request with a CA key:

openssl x509 -trustout -signkey ca.key -days 365 -req -in ca.csr -out trustedcert.pem


B. Enable TLS in SMTP Prevent via Enforce

1. To "Configure"
     a. Set Keystore Password
     b. Select "Reflect" mode
2. To "Server Settings"
     a. RequestProcessor.AllowExtensions add "STARTTLS" to the whitelist
     b. RequestProcessor.ServerSocketPort (as IronMail required)
     c. RequestProcessor.MTAResubmitPort (as IronMail required)

C. Detection Server Email Prevent (Windows)

1. Download the (downstream) MTA's cert (e.g. trustedcert.pem) to SMTP Prevent's local directory (say c:\mtacert)
2. Create a keystore and self-signed cert
     a. Use the keytool from c:\SymantecDLP\jre\bin folder
     b. Make sure 'protect' has read permissions on the keystore
     c. Run command

keytool -genkey -alias smtp_prevent -keystore c:\program files\Symantec\DataLossPrevention\DetectionServer\[YourVersion]\Protect\keystore\prevent.ks

3. Import downstream MTA trusted certificate (.pem format) into a Java Keystore as below:

keytool -importcert -keystore c:\SymantecDLP\Protect\keystore\prevent.ks -alias MTA -file c:\mtacert\trustedcert.pem


D. Recycle the SMTP Prevent server, and test the TLS mailing process and collect the logs send them back for further review by Symantec engineering.


In cases where Exchange and Message Labs have been involved, you may need to try the following:

Messages are coming from and Exchange 2007 server to SMTP Prevent (v10) and then forwarding mail to Message Labs. There is a requirement that mail be transmitted via TLS until it gets to Message Labs (from there it is opportunistic).

The MTA integration guide states that the Prevent server has to have the Public key certificate from the downstream MTA in its keystore. Message Labs sends its public key as part of the handshake.

Email Prevent validates the public key that it has in its own keystore against the public key that gets transmitted to it when it starts the TLS handshake. If they don’t match, Email Prevent terminates the connection. 

You must import the Public key from the downstream (forward) MTA into the Prevent keystore for TLS to work.

Within Sendmail environments ( Sentrion ), the following settings have to be disabled in the file.

ServerSSLOptions=SSL_OP_NO_SSLv2 Client

Otherwise sendmail will not conform fully to RFC 2246. Since SMTP Prevent is a TLSv1 compliant SMTP Proxy it would fail to establish a connection since SSLv2 is required to negotiate TLSv1.