We are trying to transfer files with XCOM using IBM System SSL and on the remote side XCOM uses OpenSSL, and the transfers are failing with message:
System SSL Function gsk_secure_socket_init: RC = 410: Reason = SSL message format is incorrect
The connection was traced and we can see the message below when the server certificate needs to be validated:
-> 0.102 298 TLS1.2: HSHK( CLIENT_HELLO )
<- 0.083 1400 SSL3: HSHK( SERVER_HELLO CERTIFICATE...
<- <0.001 438 Ack Psh Win=8183 RelSeq=1361 RelAck=259
-> <0.001 40 Ack Psh Win=4041 RelSeq=259 RelAck=1759
-> <0.001 47 SSL3: ALERT( FATAL: ILLEGAL_PARAMETER )
<- 0.003 56 Ack Psh Win=8191 RelSeq=1759 RelAck=266
-> 0.003 56 Ack Psh Win=4095 RelSeq=266 RelAck=1775
-> <0.001 40 Ack Psh Fin Win=4095 RelSeq=282 RelAck=1775
<- <0.001 40 Ack Psh Win=8191 RelSeq=1775 RelAck=283
<- <0.001 40 Ack Psh Fin Win=8191 RelSeq=1775 RelAck=283
-> <0.001 40 Ack Psh Win=4095 RelSeq=283 RelAck=1776
The XCOM using IBM System SSL is set with value "ALL" in the SYSconfigSSL.cnf file:
INITIATE_SIDE = ALL
RECEIVE_SIDE = ALL
Release : 12.0
Component : XCOM Data Transport for z/OS
The SystemSSL transfer fails the handshake because the remote XCOM. that is using OpenSSL, is not strictly following the SSLv3 protocol described in RFC 6101 which can be found at: https://datatracker.ietf.org/doc/html/rfc6101.
During the handshake, the partner sends a "certificate request" message. By this message, the 'server' part of the connection (the one who received the connection request) asks the 'client'
part to send its certificate.
The trace on the System SSL system shows that the message includes a "Supported Cert Types" section but is missing the 'distinguished names' section. We know because it says
"Distinguished Names Length = <0000>" meaning that the message contains no distinguished names.section, which is a required section in the "certificate request" message as per the
SSLv3 protocol described in rfc 6101
SystemSSL responds to that incorrect message aborting the handshake with an SSL alert 47 (hex 2F, SSL3_AD_ILLEGAL_PARAMETER) so the handshake fails.
This is a problem with OpenSSL itself, the way around that is by specifying RECEIVE_SIDE=NO in the VERIFY_CERTIFICATE section of the configssl.cnf file. Again, that is something the site would have to decide because certificate verification will be disabled.
OpenSSL has been deprecated for XCOM r12. That is why IBM System SSL was introduced with XCOM r12. We also added AT-TLS support and that will be supporting TLS 1.3 above. See XCOM Release Notes.