Fatal Alert received: Handshake Failure

Products

• CA API Gateway

Issue/Introduction

When connecting to a backend server attempting to forward a transaction from the Layer 7 Gateway, we get the following error: 
Problem routing to https://<HOSTNAME>/. Error msg: Unable to obtain HTTP response from https://<HOSTNAME>: Fatal Alert received: Handshake Failure. 

Environment

All supported versions of the API Gateway

Cause

Through a tcpdump, it was observed that the handshake failure was coming after a Client Hello to the server. Checking the allowed ciphers on the backend server and comparing them to the ciphers in Policy Manager, it was determine that there was a mismatch.

Resolution

Change the cipher suites on the backend server to match the ones configured in the Policy Manager.

Cipher suites are configured in Policy Manager under Manage Listen Ports -> Properties -> SSL/TLS Settings

Additional Information

The Client Hello command lets the server know the following:

• The version of SSL that the client is trying to use for negotiating with the server
• Some random bytes generated by the client that will be used next to generate a master key for encryption.
• A list of encryption algorithms called cipher suites. The client tells the server which cipher suites it understands.
• A list of compression algorithms supported by the client. If one of them is selected by the server the algorithm will be used to compress each message.
• Optionally: A list of extensions that can be used to improve the security of the handshake. These extensions are not part of the protocol itself but, if the server understands the extensions, they will be used.

Here is a Linux test script that can be run to test the cipher suites being used by a server: 

#!/usr/bin/env bash

# OpenSSL requires the port number.

SERVER=SERVER_IP:443
DELAY=1
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')

echo Obtaining cipher list from $(openssl version).

for cipher in ${ciphers[@]}
do
echo -n Testing $cipher...
result=$(echo -n | openssl s_client -cipher "$cipher" -connect $SERVER 2>&1)

if error:" ; then
  error=$(echo -n $result | cut -d':' -f6)
  echo NO \($error\)
else
  if | "$result" =~ "Cipher    :" ; then
    echo YES
  else
    echo UNKNOWN RESPONSE
    echo $result
  fi
fi

sleep $DELAY

done