How to setup TLS encryption and what logs to set if further troubleshooting is required
Make sure that the following settings are set for proper logging on the SMTP Prevent.
Set the log levels on SMTP Prevent
Within the SymantecDLP/Protect/config directory adjust the file RequestProcessorLogging.properties and set or assign the following value
com.vontu.mta.log.SmtpOperationalLogHandler.level = FINEST
java.util.logging.FileHandler.level = FINEST com.vontu.mta.rp.level = FINEST
com.vontu.mta.rpl.level = FINEST
Then restart SMTP Prevent
(1) Please verify that the wording of "STARTTLS" has presented in the "RequestProcessor.AllowExtensions" whitelist.
If not, add it in, and recycle the SMTP prevent server and try it again.
To edit, log into the DLP Console as Administrator and go to
System > Servers > Overview > Click on the SMTP Prevent Server > Click on 'Server Settings'
(2) If above still not working, then the problem may relate to the format of the certificate that imported from MTA.
Proper X509 certificates need to be installed on prevent in order for this to work securely.
(In this POA - ie. trustedcert.pem) .pem file -- Privacy Enhanced Mail Certificate
Follow steps below for TLS setup
A. Down Stream MTA
1. Generate a new public/private key pair:
openssl req -new -key ca.key -out ca.csr
2. Generate a .pem file by signing a certificate request with a CA key:
openssl x509 -trustout -signkey ca.key -days 365 -req -in ca.csr -out trustedcert.pem
B. Enable TLS in SMTP Prevent via Enforce
1. To "Configure"
a. Set Keystore Password
b. Select "Reflect" mode
2. To "Server Settings"
a. RequestProcessor.AllowExtensions add "STARTTLS" to the whitelist
b. RequestProcessor.ServerSocketPort (as IronMail required)
c. RequestProcessor.MTAResubmitPort (as IronMail required)
C. SMTP Prevent (Windows)
1. Download the (downstream) MTA's cert (e.g. trustedcert.pem) to SMTP Prevent's local directory (say c:\mtacert)
2. Create a keystore and self-signed cert
a. Use the keytool from c:\SymantecDLP\jre\bin folder
b. Make sure 'protect' has read permissions on the keystore
c. Run - keytool -genkey -alias smtp_prevent -keystore c:\SymantecDLP\Protect\keystore\prevent.ks
3. Import downstream MTA trusted certificate (.pem format) into a Java Keystore
a. Run - keytool -importcert -keystore c:\SymantecDLP\Protect\keystore\prevent.ks -alias MTA -file c:\mtacert\trustedcert.pem
D. Recycle the SMTP Prevent server, and test the TLS mailing process and collect the logs send them back for further review by Symantec engineering.
In cases where Exchange and Message Labs have been involved, you may need to try the following:
Messages are coming from and Exchange 2007 server to SMTP Prevent (v10) and then forwarding mail to Message Labs. There is a requirement that mail be transmitted via TLS until it gets to Message Labs (from there it is opportunistic).
The MTA integration guide states that the Prevent server has to have the Public key certificate from the downstream MTA in its keystore. Message Labs sends its public key as part of the handshake.
Email Prevent validates the public key that it has in its own keystore against the public key that gets transmitted to it when it starts the TLS handshake. If they don’t match, Email Prevent terminates the connection.
You must import the Public key from the downstream (forward) MTA into the Prevent keystore for TLS to work.
Within Sendmail environments ( Sentrion ), the following settings have to be disabled in the Sendmail.cf file.
Otherwise sendmail will not conform fully to RFC 2246. Since SMTP Prevent is a TLSv1 compliant SMTP Proxy it would fail to establish a connection since SSLv2 is required to negotiate TLSv1.