Setting up TLS encryption for SMTP Prevent

Products

• Data Loss Prevention Network Prevent for Email
• Data Loss Prevention

Issue/Introduction

How to setup TLS encryption, and what logs to set if further troubleshooting is required.

Resolution

To enable Debug logging for troubleshooting, please follow the instructions in the following KB article...
https://knowledge.broadcom.com/external/article/160811

 

Enable TLS in SMTP Prevent via Enforce:
- Open the Enforce Console
     - Go to the Server Overview Page
          - System > Servers and Detectors > Overview
     - Open the server you wish to configure
          - Click on the 'Configure' button
               - Select the 'Inline SMTP' tab
                    -
                    - Make sure to disable 'Trial Mode' if you wish to block traffic
                    - Set the 'Keystore Password' in the corresponding field
                    - Select the appropriate option for the Next Hop Configuration (Reflect/Forward).
                    - Select 'Save' to save your changes and go back to the Detection Server Overview Page.
          - Click on the 'Server Settings' button
               - Find: 'RequestProcessor.AllowExtensions' and add "STARTTLS" to the end of the field (do not remove the previous values)
               - Find: 'RequestProcessor.ServerSocketPort' and set to the appropriate value. This is the listening port for the DLP Detection Server.
               - Find: 'RequestProcessor.MTAResubmitPort' and set to the appropriate value. This should be the port your downstream MTA is listening on. 
               - Note: ServerSocketPort and MTAResubmitPort can only use the same port in Forwarding Mode. In Reflect Mode, these ports MUST be different.

 

Locations of Keytool vary with version of DLP and OS of your servers:

Windows:

• Default Path: C:\Program Files\AdoptOpenJRE\jdk8u[version]-jre

Linux:

• Default Path: /opt/AdoptOpenJRE/jdk8u[version]-jre

 

Use Keytool to create the SMTP Certificate (this will automatically create the Keystore):
     - Download the (downstream) MTA's cert (e.g. trustedcert.pem) to SMTP Prevent's local directory (say c:\mtacert)
     - Create a keystore and self-signed cert
          - Use the keytool from DLP (see keytool locations below)
          - Make sure 'protect' has read permissions on the keystore
          - Run command

Windows: keytool -genkey -alias smtp_prevent -keystore C:\ProgramData\Symantec\DataLossPrevention\DetectionServer\[DLPVersion]\keystore\prevent.ks

Linux: keytool -genkey -alias smtp_prevent -keystore /var/Symantec/DataLossPrevention/DetectionServer/[DLPVersion]/keystore/prevent.ks

 

- Import the downstream MTA certificate into the Java Keystore as below:

Windows: keytool -importcert -keystore c:\ProgramData\Symantec\DataLossPrevention\DetectionServer\[DLPVersion]\keystore\prevent.ks -alias [YourAlias] -file [FullPathToSMTPCertificate]

Linux: keytool -importcert -keystore
/var/Symantec/DataLossPrevention/DetectionServer/[DLPVersion]/keystore/prevent.ks -alias [YourAlias] -file [FullPathToSMTPCertificate]

     - Recycle the SMTP Prevent server, and test the TLS mailing process
     - Collect the logs and send them back to support for further review if you are experiencing any problems.

Note: For Windows users, you can use the script from the following KB article to create/import your certificates making this process significantly easier...
https://knowledge.broadcom.com/external/article/273794

 

 

 

Additional Information

In cases where Exchange and Message Labs have been involved, you may need to try the following:

Messages are coming from and Exchange 2007 server to SMTP Prevent (v10) and then forwarding mail to Message Labs. There is a requirement that mail be transmitted via TLS until it gets to Message Labs (from there it is opportunistic).

The MTA integration guide states that the Prevent server has to have the Public key certificate from the downstream MTA in its keystore. Message Labs sends its public key as part of the handshake.

Email Prevent validates the public key that it has in its own keystore against the public key that gets transmitted to it when it starts the TLS handshake. If they don’t match, Email Prevent terminates the connection. 

You must import the Public key from the downstream (forward) MTA into the Prevent keystore for TLS to work.

Note:
Within Sendmail environments ( Sentrion ), the following settings have to be disabled in the Sendmail.cf file.

ServerSSLOptions=SSL_OP_NO_SSLv2 Client
SSLOptions=SSL_OP_NO_SSLv2

Otherwise sendmail will not conform fully to RFC 2246. Since SMTP Prevent is a TLSv1 compliant SMTP Proxy it would fail to establish a connection since SSLv2 is required to negotiate TLSv1.