XCOMU0298E "return value = 9" and XCOMU0780E "SSL23_GET_SERVER_HELLO:unknown protocol"

Products

• XCOM Data Transport
• XCOM Data Transport - Linux PC

Issue/Introduction

RHEL Linux 7.9 with new install of XCOM 11.6 SP01 GA i.e. r11.6 16083 SP01

 

A local loopback test with "xcomtcp -ping" using secure port 8045 is failing

 

xcomtcp -ping REMOTE_SYSTEM=127.0.0.1 PORT=8045 SECURE_SOCKET=YES TRNENCRL_CIPHER=ALL

#XCOMU0298E Unable to allocate remote transaction program: Txpi  215: Socket send error return value = 9

The xcom.log shows:

XCOMU0780E Txpi  308: TxpiInitSSL Failed msg = <error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol> value = 4294967295:

 

The XCOM 8044 and 8045 ports hosted by xinetd are active and listening: 

netstat -an | grep 804

tcp        0      0 0.0.0.0:8044            0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:8045            0.0.0.0:*               LISTEN

tcp6       0      0 :::8046                 :::*                    LISTEN

tcp6       0      0 :::8047                 :::*                    LISTEN

An unsecure loopback test using port 8044 is also working:

xcomtcp -ping REMOTE_SYSTEM=127.0.0.1 PORT=8044 SECURE_SOCKET=NO TRNENCRL_CIPHER=ALL

XCOMU0882I PING INFO FOR 127.0.0.1

XCOMU0882I RELEASE=r11.6   SP01 GEN LEVEL 16083    SYSTEM NAME=server1 SYSTEM ID=LINX

XCOMU0882I NEGOTIATED CIPHER=XCOM

Environment

Component: XCOM Data Transport for Linux
Release: Any

Resolution

Support could not recreate the problem with the same XCOM 11.6 16083 SP01 level and also using the same xcom.glb, xcom.cnf and configssl.cnf files.
It was suggested to run this openssl test:
openssl s_client -connect 127.0.0.1:8045 -showcerts -status -msg > openssl.out 2>&1

In the openssl.out file the SSL handshake showed that after the client hello, the server hello immediately failed with the same error as show in the xcom.log file:
**********
CONNECTED(00000003)
>>> TLS 1.2  [length 0005]
    16 03 01 01 25
>>> TLS 1.2 Handshake [length 0125], ClientHello
    01 00 01 21 03 03 66 7b 3b a0 3f 76 8e 7b 0f 13
...
    00 0f 00 01 01
140014278428560:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
...
**********

Interestingly Support received the exact same openssl output when the xcomd is not started.
Although "netstat -an | grep 804" shows xinetd is started & listening on 8045 and the unsecure  "xcomtcp -ping" for port 8044 is working, there may be some problem with the xcomd process causing the SSL handshake to fail.
After restarting xcomd the problem was resolved.