Symantec DLP Network Prevent for E-mail fails TLS handshake with WebSense as downstream MTA

Products

• Data Loss Prevention Network Prevent for Email

Issue/Introduction

When Network Prevent for E-mail detection server is integrated with WebSense MTA as downstream MTA and TLS is enabled in the communication, it's possible that the TLS handshake will fail with the following errors in the RequestProcessor logs:

 

Nov 15, 2018 11:19:39 AM com.vontu.mta.rp.ESMTPRequestProcessorThread process

SEVERE: RPT(1e): Returning fatal response and terminating connections due to unhandled exception.

java.lang.RuntimeException: Could not generate DH keypair

                at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1429)

                at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)

                at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)

                at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

                at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

                at com.vontu.mta.rp.tls.TLSEngine.handshakeUnwrap(TLSEngine.java:356)

                at com.vontu.mta.rp.tls.SecureESMTPPeer.recvHandshakeData(SecureESMTPPeer.java:437)

                at com.vontu.mta.rp.tls.SecureESMTPPeer.recv(SecureESMTPPeer.java:570)

                at com.vontu.mta.rp.ESMTPRequestProcessorThread.readPeer(ESMTPRequestProcessorThread.java:971)

                at com.vontu.mta.rp.ESMTPRequestProcessorThread.process(ESMTPRequestProcessorThread.java:1061)

                at com.vontu.mta.rp.ESMTPRequestProcessorThread.run(ESMTPRequestProcessorThread.java:1550)

                at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.RuntimeException: Could not generate DH keypair

                at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:142)

                at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:114)

                at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:708)

                at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:268)

                at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

                at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)

                at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)

                at java.security.AccessController.doPrivileged(Native Method)

                at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)

                at com.vontu.mta.rp.tls.TLSEngine.runDelegatedTasks(TLSEngine.java:406)

                at com.vontu.mta.rp.tls.TLSEngine.handshakeUnwrap(TLSEngine.java:389)

                ... 6 more

Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 2048 (inclusive)

                at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DHKeyPairGenerator.java:120)

                at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:674)

                at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:128)

                ... 16 more

Cause

This is caused by WebSense trying to use a Diffie-Hellmann key larger than 2048 bytes, which is not accepted by Java used by the NPE. 

Resolution

To address the problem, on the NPE go to the following location (change the root folder if it's different than the default):

C:\SymantecDLP\jre\bin\security\java.security (15.0 and earlier)

C:\Program Files\Symantec\Data Loss Prevention\Server JRE\1.8.0_162\lib\security (15.1)

C:\Program Files\Symantec\DataLossPrevention\ServerJRE\1.8.0_162\lib\security (15.5)

Then open the file java.security and change the following line from (note that the line can be different in each environment):

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 1024, EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC

to:

jdk.tls.disabledAlgorithms=DH, SSLv3, RC4, MD5withRSA, EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC

The goal is to add DH to the list of disabled algorithms to Java configuration on NPE, so that DH is not used at all during the TLS handshake with the WebSense MTA. 

With DH disabled on NPE, the handshake should now work successfully.