What are the requirements for SMTP Prevent in order to establish TLS encrypted communication with the MTA?
SMTP Prevent acts as an RFC-2821 compliant SMTP proxy which in general extends RFC 821.
This means the generated email has to be compliant to RFC 2821 and subsequently RFC 821.
See as reference the documentation Symantec_DLP_10.0_Email_Prevent_MTA_Integration_Guide.pdf on chapter 3 (page 17) and TECH219955.
It is fully compliant to RFC 2246 ( TLSv1 ) if TLS communication is to be established with the MTA.
It may fail to establish a connection with MTAs that may not allow SSLv2 handshakes since SSLv2 is required to negotiate TLSv1.
Once the handshake has been performed the communication will occur via TLSv1.
Within Sendmail environments ( Sentrion ) , the following settings have to be disabled in the Sendmail.cf file.
If these are not disabled, the MTA-SMTP Prevent handshake would fail to establish a connection, since SSLv2 communication is initially required to negotiate TLSv1.
Per RFC 2246 :
TLS 1.0 clients that support SSL Version 2.0 servers must send SSL
Version 2.0 client hello messages [SSL2]. TLS servers should accept
either client hello format if they wish to support SSL 2.0 clients on
the same connection port. The only deviations from the Version 2.0
specification are the ability to specify a version with a value of
three and the support for more ciphering types in the CipherSpec.
Warning: The ability to send Version 2.0 client hello messages will be
phased out with all due haste. Implementors should make every
effort to move forward as quickly as possible. Version 3.0
provides better mechanisms for moving to newer versions.
In Symantec DLP V10.5 there is a bug related to handling client hello extension. See TECH221239 for details.
This bug does not occur in V11
SMTP Prevent is TLSv1 compliant per RFC 2246, but does not comply to the following RFCs as they apply to higher TLS versions.
RFC 4346 is for TLS 1.1
RFC 5246 is for TLS 1.2