If Encryption Management Server is configured to proxy mail to an MTA that issues a Certificate Request as part of the TLS handshake, the TLS connection cannot be established after the first message is proxied.
The only well known MTA that always seems to issue a Certificate Request is Microsoft Exchange Online (neither Gmail or Symantec email security.cloud do this). Therefore, this issue is only likely to occur if Encryption Management Server is configured to proxy directly to Exchange Online.
The example Mail log below shows the first message SMTP-00000 being accepted but the second message SMTP-00001 failing and being retried as SMTP-00002. The error is:
the TLS connection was unexpectedly shutdown
As described in article 213702, as part of the TLS handshake, a remote MTA may issue a Certificate Request. Encryption Management Server does not support such requests and therefore responds with a zero length certificate. This complies with section 7.4.6 of RFC 5246.
However, the mail proxy service of Encryption Management Server 3.4.2 MP2 and above only responds with a zero length certificate once. In subsequent connections it responds with the certificate chain of its certificate.
For example, here is a tcpdump capture of the TLS handshake from a first message. Entry 17 shows the Certificate Request by the remote host. Entry 18 shows Encryption Management Server responding correctly with a zero length certificate:
On subsequent connections, however, Encryption Management Server responds with the certificate chain - the root and intermediate certificate(s) - of its server certificate and the TLS handshake fails:
Symantec Encryption Management Server 3.4.2 MP2, 3.4.2 MP3, 3.4.2 MP4, 3.4.2 MP5, 10.5 and 10.5 MP1.
Upgrade to release 10.5 MP2 or above.
If you cannot upgrade, there are several workarounds for this issue.
The simplest workaround for this issue is to use a self-signed certificate on the mail proxy interface that connects to the remote MTA. To create a self-signed certificate do the following:
A certificate issued either by an internal or external Certificate Authority can be effectively made into a self-signed certificate by deleting the certificates in its certificate chain:
If Outbound mail is being proxied to an MTA that issues a Certificate Request, you can configure Encryption Management Server to use the internal postfix service to relay mail. Postfix always responds to Certificate Requests with a zero length certificate:
If Inbound mail is being proxied to an MTA that issues a Certificate Request, you cannot use the internal postfix service because there is no Send mail directly to recipient mailserver setting for Inbound mail.
In this scenario, you need to configure Encryption Management Server to proxy to an MTA that does not issue a Certificate Request.
If you do not have a suitable MTA available then you can install another Encryption Management Server and use it for this purpose. Note that the new Encryption Management Server cannot be clustered with the existing Encryption Management Server, it must be standalone.
To configure a standalone Encryption Management Server to relay mail, do the following:
To configure the existing Encryption Management Server to use the new server, do the following:
Inbound mail will now be proxied to the standalone Encryption Management Server which will use its internal postfix service to deliver the mail to the MTA that issues Certificate Requests.