SSLV3 Connections will Fail when FIPS Mode is Enabled and when SMTP Authentication has "Require TLS encryption" Disabled

Products

• Messaging Gateway

Issue/Introduction

• FIPS mode is enabled on an Symantec Messaging Gateway (SMG) host.
  To verify this follow these steps:
  1. Login to the CLI command prompt on the SMG scanner host
  2. At the prompt enter the following command and press enter
       fipsmode status
  3. If the result is FIPS mode, FIPS is enabled on the scanner host
   
• SMG host is configured to have "Require TLS encryption" is disabled
  To verify this follow these steps:
  1. Login to the SMG Brightmail Control Center (BCC) web UI
  2. Click on Administration->Hosts->Configuration
  3. Select the affected scanner host and press the Edit Button
  4. Click on the SMTP tab
  5. Click on the Authentication sub-menu tab
  6. In the Authentication Mail Settings section, having the "Require TLS encryption" check-box not having a check mark, indicates this condition.
   
• SSLv3 connections fail to the SMG host.
  To verify this follow these steps:
  1. On a server where openSSL is installed that is not the local scanner host
  2. Connect to the scanner host using the following command:
  openssl s_client -connect <host.scanner.fqdn>:25 -starttls smtp -ssl3
   
  NOTE: in the command above replace <host.scanner.fqdn> with the fully qualified domain name of the scanner host, i.e. scanner01.company.com
  
  3. Seeing an error message similar to the following indicates this condition:
        Loading 'screen' into random state - done
        CONNECTED(0000078C)
        3400:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:.\ssl\s3_pkt.c:534:

 

Cause

This is expected behavior. Symantec Messaging Gateway does not support SSLv3 connections when FIPS mode is enabled.
 

Resolution

When FIPS mode is turned on, even if the Require TLS encryption option is disabled, connections using SSLv3.0 and earlier are not supported. This is as per FIPS 140-2 level 1 requirements. Please see the Symantec Messaging Gateway FIPS 140-2 level 1 Deployment Guide for more information.