How does XCOM validate client/server certificates in SSL transfers
search cancel

How does XCOM validate client/server certificates in SSL transfers

book

Article ID: 398313

calendar_today

Updated On:

Products

XCOM Data Transport XCOM Data Transport - Linux PC XCOM Data Transport - Windows XCOM Data Transport - z/OS

Issue/Introduction

This question is generic to all XCOM platforms for SSL/secure transfers.

In our environment the server SSL certificates can be shared across multiple XCOM server hosts i.e. the certificate validation appears to work when something like a Common Name Mismatch error would be expected. This is a cyber security risk.
What is the purpose of a server certificate if it is not matching-up the hostname or IP or DNS (even in a SAN certificate). It appears only the root and intermediate (Chain) certificates actually matter.
Is there a simple configuration change to enforce this check?

In what follows please take note of the following points:

Regarding the sample certificates which can be created by the supplied makeca, makeclient and makeserver scripts:
They are in no way fit for a secure production environment. They are provided to get SSL transfers to work with minimal effort. As such, all the fields in the certificates are meaningless for a specific implementation, for example the DNS names given in the cassl.conf, clientssl.conf and serverssl.conf could not possibly reflect an actual user environment. The sample client and server certificates could be used on any machine only validating a common root certificate. Proper configuration of a secure production environment is beyond the scope of this article.

Regarding the configssl.cnf file:
The default values given validate the certificates but turn off any verification of given fields in the certificate like VERIFY_MACHINE, SERIAL_NUMBER, NAME, TITLE etc.

As an example, this article shows the possible verifications for just one field, VERIFY_MACHINE, client side.

Environment

XCOM™ Data Transport® for UNIX/Linux
XCOM™ Data Transport® for Windows
XCOM™ Data Transport® for z/OS
XCOM™ Data Transport® for IBM i AS/400
XCOM™ Data Transport® for HP NonStop

Resolution

Table of Contents


XCOM SSL certificate validation

Looking at XCOM for Linux and the default $XCOM_HOME/config/configssl.cnf file:

The VERIFY_CERTIFICATE parameter is the overriding parameter controlling all certificate checking.
By default it is set to YES i.e. the configssl.cnf shows:
# Mandatory, YES/NO
[VERIFY_CERTIFICATE]
INITIATE_SIDE = YES
RECEIVE_SIDE  = YES 

Other relevant parameters:
The VERIFY_MACHINE parameter which defaults to NO i.e. 
# Mandatory, YES/NO
[VERIFY_MACHINE]
INITIATE_SIDE = NO
RECEIVE_SIDE  = NO

The optional HOST_NAME parameter:
*****
# Optional, matches against the Subject Alternative Name DNS: field in the certificate
# The wildcard character can be used in the Subject Alternative Name
# HOST_NAME can contain one or more INITIATE_SIDEx and RECEIVE_SIDEx (x=1, 2, 3, etc.)
# INITIATE_SIDEx host name matches the certificate sent by the receive side
# RECEIVE_SIDEx host name matches the certificate sent by the initiate side
# example: INITIATE_SIDE1 = hostname.aa.com or INITIATE_SIDE1 = *name.aa.com 
#          RECEIVE_SIDE1 = *.aa.com
[HOST_NAME]
INITIATE_SIDE1 =
RECEIVE_SIDE1  =
*****

These checks apply.

  1. If VERIFY_MACHINE is set to YES that results in a "dynamic" reverse DNS lookup of the partner hostname from the received partner IP address and that will be checked against the partner certificate DNS name. This might be the simplest solution in terms of having a generic configssl.cnf file across all of them as long as VERIFY_MACHINE=YES is set i.e. don't have to have individual files with different HOST_NAME values.
    However if any transfers are received via a proxy the VERIFY_MACHINE may not work because the reverse DNS lookup of the received IP will not give the correct hostname of the originating partner XCOM server to do the check against that partner's certificate DNS name. In that scenario using HOST_NAME would be an alternative option.

  2. If the HOST_NAME section has any data it will be checked against the partner certificate DNS name (does not require VERIFY_MACHINE to be set to YES).

 

Internal tests

Using a Windows machine to initiate a SSL transfer and a Linux as the target, XCOM Engineering performed tests to confirm the VERIFY_MACHINE behaviour as follows:

  1. The defaults from Windows configssl.cnf:
    [VERIFY_CERTIFICATE]
    INITIATE_SIDE = YES
    RECEIVE_SIDE  = YES

    [VERIFY_MACHINE]
    INITIATE_SIDE = NO
    RECEIVE_SIDE  = NO

    Using certificates created with XCOM makeca, makeclient and makeserver scripts a SSL transfer works.


  2. Change Windows configssl.cnf:
    [VERIFY_MACHINE]
    INITIATE_SIDE = YES
    RECEIVE_SIDE  = NO

    On the Windows side the transfer fails. The trace shows:
    17:46:00.022 P43412 T43096   TCPcu catoossl 3707: peer_certificate:
    17:46:00.022 P43412 T43096   TCPcu catoossl 3719: peer_certificate: pSSLhostname.
    17:46:00.022 P43412 T43096   TCPcu catoossl 3786: peer_certificate: pTxpiSSLParms->pSSLhostname = 0000000000000000.
    17:46:00.022 P43412 T43096   TCPcu catoossl 3788: peer_certificate: pSSLverifymachine.
    17:46:00.022 P43412 T43096   TCPcu catoossl 3797: peer_certificate: Name(DNS) <linux.broadcom.com> value(cert) <serverssl @ca.com>.
    17:46:00.022 P43412 T43096   TCPcu catoossl 3797: peer_certificate: Name(DNS) <linux.broadcom.com> value(cert) <hostname.ca.com>.
    17:46:00.022 P43412 T43096   TCPcu catoossl 3808: peer_certificate: ret_ok = 0 X509_free.
    17:46:00.022 P43412 T43096   TCPcu catoossl 3810: Error 316
    17:46:00.022 P43412 T43096 > TCPcu catoossl 3810: Txpi  316: TxpiSSL peer certificate: Certificate verification failed (DNS verify machine name mismatch) error 4

    XCOM Windows uses the IP address of the incoming transfer to do a reverse DNS lookup and finds linux.broadcom.com. It cannot match it against the Linux server
    certificate which has "serverssl @ca.com" and a second possibility "hostname.ca.com".

    Recreate the Linux server certificate with a change in "serverssl.conf", section [ req_distinguished_name ]:
    emailAddress                    = linux.broadcom.com

    The transfer is successful and the trace shows:
    19:47:17.616 P40400 T24736   TCPcu catoossl 3788: peer_certificate: pSSLverifymachine.
    19:47:17.616 P40400 T24736   TCPcu catoossl 3797: peer_certificate: Name(DNS) < linux.broadcom.com > value(cert) < linux.broadcom.com >.

Additional Information

This is currently detailed under XCOM for z/OS documentation but not under XCOM for distributed platforms documentation : XCOM™ Data Transport® for z/OS 12.0 > Installing > Complete Configuration Tasks > Implement SSL/TLS > Configure the System SSL Configuration File

The following SSL documentation for other platforms is being currently reviewed for further improvements:
XCOM Data Transport for UNIX/Linux 11.6.1 > Using > Generating TLS/SSL Certificates
XCOM Data Transport for UNIX/Linux 12.0 > Installing > Complete Configuration Tasks > Set Up TLS/SSL Mode
XCOM™ Data Transport® for Windows 12.0 > Installing > Complete Configuration Tasks > Set Up TLS/SSL Mode
XCOM™ Data Transport® for AS/400 11.0 > Administrating > Generating TLS/SSL Certificates
XCOM™ Data Transport® for HP NonStop 11.1 > Administrating > Generating TLS/SSL Certificates

Powered by Wolken Software