Encryption Management Server cannot connect over SMTP TLS if the remote host issues a Certificate Request

Products

• Encryption Management Server
• Encryption Management Server Powered by PGP Technology
• Gateway Email Encryption
• Gateway Email Encryption Powered by PGP Technology

Issue/Introduction

If Encryption Management Server is configured to proxy mail to an MTA that issues a Certificate Request as part of the TLS handshake, the TLS connection cannot be established after the first message is proxied.

The only well known MTA that always seems to issue a Certificate Request is Microsoft Exchange Online (neither Gmail or Symantec email security.cloud do this). Therefore, this issue is only likely to occur if Encryption Management Server is configured to proxy directly to Exchange Online.

The example Mail log below shows the first message SMTP-00000 being accepted but the second message SMTP-00001 failing and being retried as SMTP-00002. The error is:

the TLS connection was unexpectedly shutdown

Environment

Symantec Encryption Management Server 3.4.2 MP2, 3.4.2 MP3, 3.4.2 MP4, 3.4.2 MP5, 10.5 and 10.5 MP1.

Cause

As described in article 213702, as part of the TLS handshake, a remote MTA may issue a Certificate Request. Encryption Management Server does not support such requests and therefore responds with a zero length certificate. This complies with section 7.4.6 of RFC 5246.

However, the mail proxy service of Encryption Management Server 3.4.2 MP2 and above only responds with a zero length certificate once. In subsequent connections it responds with the certificate chain of its certificate.

For example, here is a tcpdump capture of the TLS handshake from a first message. Entry 17 shows the Certificate Request by the remote host. Entry 18 shows Encryption Management Server responding correctly with a zero length certificate:

On subsequent connections, however, Encryption Management Server responds with the certificate chain - the root and intermediate certificate(s) - of its server certificate and the TLS handshake fails:

Resolution

Upgrade to release 10.5 MP2 or above.

If you cannot upgrade, there are several workarounds for this issue.

1. Use a self-signed certificate

The simplest workaround for this issue is to use a self-signed certificate on the mail proxy interface that connects to the remote MTA. To create a self-signed certificate do the following:

1. In the administration console, navigate to System / Network and click on the Certificates button.
2. Click on the Add Certificate button.
3. In the Hostname field, enter the CN (Common Name) of the certificate. For example, keys.example.com.
4. From the Key Size dropdown list, select 2048.
5. Optionally, from the Expiration dropdown list, select a number of years other than the default of 1 year.
6. Leave the Contact Email field empty.
7. Optionally, complete some or all of the remaining fields.
8. Click the Generate Self-Signed button to generate the self-signed certificate.
9. Navigate to System / Network.
10. Select the Interface that connects to the mail proxy.
11. Assign the self-signed certificate to the interface by selecting it from the Assigned Certificate dropdown list.
12. Click the Save  button.

2. Delete the certificate chain of a certificate that is not self-signed

A certificate issued either by an internal or external Certificate Authority can be effectively made into a self-signed certificate by deleting the certificates in its certificate chain:

1. In the administration console, navigate to Keys / Trusted Keys.
2. Search for the certificate's root certificate.
3. Click on the Delete button to delete it.
4. Delete all the Intermediate certificates in the same way.

3. Relay through Postfix for Outbound mail

If Outbound mail is being proxied to an MTA that issues a Certificate Request, you can configure Encryption Management Server to use the internal postfix service to relay mail. Postfix always responds to Certificate Requests with a zero length certificate:

1. In the administration console, navigate to Mail / Proxies.
2. Click on the Outbound or Unified proxy entry.
3. Ensure that the setting Send mail directly to recipient mailserver is enabled, not Send all outbound mail to relay (Unified proxy) or Proxy mail to SMTP server (Outbound proxy):
   
4. Navigate to Mail / Mail Routes.
5. Ensure that the wildcard * entry is configured to use the MTA that is issuing the Certificate Request. All mail for domains that are not explicitly listed under Mail Routes will be relayed to this MTA:
   

4. Relay through another Encryption Management Server for Inbound mail

If Inbound mail is being proxied to an MTA that issues a Certificate Request, you cannot use the internal postfix service because there is no Send mail directly to recipient mailserver setting for Inbound mail.

In this scenario, you need to configure Encryption Management Server to proxy to an MTA that does not issue a Certificate Request.

If you do not have a suitable MTA available then you can install another Encryption Management Server and use it for this purpose. Note that the new Encryption Management Server cannot be clustered with the existing Encryption Management Server, it must be standalone.

To configure a standalone Encryption Management Server to relay mail, do the following:

1. In the administration console, navigate to Mail / Proxies.
2. Click on the Add Proxy button.
3. In the Port field, enter 25.
4. Optionally, from the Security dropdown list, select STARTTLS Require.
5. From the SMTP Proxy Type dropdown list select Outbound.
6. Do not change the default option of Send mail directly to recipient mailserver.
7. Click the Save button.
   
8. Navigate to Mail / Mail Routes.
9. Ensure that the wildcard * route is configured to use the MTA that issues Certificate Requests.
10. Navigate to Mail / Mail Policy.
11. Click on the Outbound policy chain.
12. Click on the Add Rule button at the bottom of the page.
13. Enter a Rule Name. For example, Relay.
14. Optionally enter a Description. For example, Relay mail using Postfix and Mail Routes.
15. Select the Condition This condition is always true.
16. Select the Action Deliver Message.
17. Click the Save button.
    
18. The new rule will be last in the list. From the dropdown, select 1 to make it first in the list. This rule will therefore be the only rule that is ever matched.

To configure the existing Encryption Management Server to use the new server, do the following:

1. In the administration console, navigate to Mail / Proxies.
2. Click on the Unified or Inbound proxy entry.
3. For Inbound mail, in the Mailserver field, enter the FQDN of the new Encryption Management Server.
4. Click the Save button.

Inbound mail will now be proxied to the standalone Encryption Management Server which will use its internal postfix service to deliver the mail to the MTA that issues Certificate Requests.

Additional Information

EPG-23361