System SSL function gsk_secure_socket_init: RC = 410: Reason = SSL message format is incorrect

Products

• XCOM Data Transport - z/OS

Issue/Introduction

We are trying to transfer files with XCOM using IBM System SSL and on the remote side XCOM uses OpenSSL, and the transfers are failing with message:

System SSL Function gsk_secure_socket_init: RC = 410: Reason = SSL message  format is incorrect.                                                     

The connection was traced and we can see the message below when the server certificate needs to be validated:
                       
 ->  0.102   298  TLS1.2: HSHK( CLIENT_HELLO )               
<-   0.083  1400  SSL3: HSHK( SERVER_HELLO CERTIFICATE...    
<-  <0.001   438  Ack Psh Win=8183 RelSeq=1361 RelAck=259    
 -> <0.001    40  Ack Psh Win=4041 RelSeq=259 RelAck=1759    
 -> <0.001    47  SSL3: ALERT( FATAL: ILLEGAL_PARAMETER )    
<-   0.003    56  Ack Psh Win=8191 RelSeq=1759 RelAck=266    
 ->  0.003    56  Ack Psh Win=4095 RelSeq=266 RelAck=1775    
 -> <0.001    40  Ack Psh Fin Win=4095 RelSeq=282 RelAck=1775
<-  <0.001    40  Ack Psh Win=8191 RelSeq=1775 RelAck=283    
<-  <0.001    40  Ack Psh Fin Win=8191 RelSeq=1775 RelAck=283
 -> <0.001    40  Ack Psh Win=4095 RelSeq=283 RelAck=1776    

The XCOM using IBM System SSL is set with value "ALL" in the SYSconfigSSL.cnf file:

[SSL_METHOD]
INITIATE_SIDE = ALL
RECEIVE_SIDE = ALL

 

Environment

XCOM™ Data Transport® for z/OS

Cause

The SystemSSL transfer fails the handshake because the remote XCOM that is using OpenSSL, is not strictly following the SSLv3 protocol described in RFC 6101 .

During the handshake, the partner sends a "certificate request" message. By this message, the 'server' part of the connection (the one who received the connection request) asks the 'client' part to send its certificate.

The trace on the System SSL system shows that the message includes a "Supported Cert Types" section but is missing the 'distinguished names' section. We know because it says "Distinguished Names Length = <0000>" meaning that the message contains no distinguished names.section, which is a required section in the "certificate request" message as per the SSLv3 protocol described in rfc 6101.

SystemSSL responds to that incorrect message aborting the handshake with an SSL alert 47 (hex 2F, SSL3_AD_ILLEGAL_PARAMETER) so the handshake fails.

Resolution

This is a problem with OpenSSL itself, the way around that is by specifying RECEIVE_SIDE=NO in the VERIFY_CERTIFICATE section of the configssl.cnf file. Again, that is something the site would have to decide because certificate verification will be disabled.

OpenSSL has been deprecated for XCOM r12. That is why IBM System SSL was introduced with XCOM r12. We also added AT-TLS support and that will be supporting TLS 1.3 above. See New Features and scroll down.