XCOM SSL ping to Windows server fails with return value 10038

Products

XCOM Data Transport XCOM Data Transport - Windows

Issue/Introduction

Attempting an XCOM SSL (secure) ping test from a working XCOM for Windows server to another problem server using port 8045 fails with return value 10038 i.e.

"%XCOM_HOME%\xcomtcp" -ping REMOTE_SYSTEM=##.##.#.# PORT=8045 SECURE_SOCKET=YES TRNENCRL_CIPHER=ALL
Copyright (c) 2012 CA. All rights reserved.
#XCOMN0298E Unable to allocate remote transaction program: Txpi 215: Socket send error return value = 10038

The unsecure port test is successful.
"%XCOM_HOME%\xcomtcp" -ping REMOTE_SYSTEM=##.##.#.# PORT=8044 SECURE_SOCKET=NO TRNENCRL_CIPHER=ALL
Copyright (c) 2012 CA. All rights reserved.
XCOMN0882I PING INFO FOR ##.##.#.#
XCOMN0882I RELEASE=r11.6 SP03 GEN LEVEL 22110 SYSTEM NAME=sys1 SYSTEM ID=w19
XCOMN0882I NEGOTIATED CIPHER=XCOM

A similar error occurs in the opposite direction SSL ping from the problem server to the working server.
Only occurs for one Windows server. Many others with an identical configuration have no problem.

Environment

XCOM™ Data Transport® for Windows
AWS is hosting the Windows servers.

Resolution

1. A telnet test from working server to problem server using port 8045 was successful which would seem to indicate no underlying firewall problem.

2. After a xcomtcp secure ping (port 8045) from working server to problem server:
a. The xcom.log on the working (sending) server showed delayed response of 19 seconds:
*****
2024/07/20 10:47:25 TID= PRG=xcomtcp PID=2324 IP=##.##.#.#:8045
XCOMN0813I Secure TCP/IP Connection Requested.

2024/07/20 10:47:44 TID= PRG=xcomtcp PID=2324 IP=##.##.#.#:8045
XCOMN0780E:
*****

b. The xcom.log on the problem (receiving) server showed just:
*****
2024/07/20 10:47:25 TID=REMOTE PRG=xcomtcp PID=7048 IP=##.##.#.#:58046
XCOMN0812I Remote Secure TCP/IP Connection Requested.
*****
Note that here is no later message "XCOMN0818I Secure TCP/IP Connection Ended" which is what should occur when there is a successful ping.

The root cause turned out to be some network configuration problem which impacted the packets being sent/received for the SSL ping/connection test.

March 2025:
Repeated occurrence of the same problem with 19 second delay using a secure xcomtcp ping (port 8045).

When use secure xcomtcp ping (port 8045) from the problem server to a working server this error occurs:
#XCOMN0298E Unable to allocate remote transaction program: Txpi 215: Socket send error return value = 10038
In the remote server xcom.log it shows this immediately after the "XCOMN0812I Remote Secure TCP/IP Connection Requested" message.
XCOMN0780E Txpi 308: TxpiInitSSL Failed msg = <error:0000000:lib-
When repeat the test using unsecure xcomtcp ping (port 8044) no problem occurs.

Detailed investigation using Wireshark showed that for secure pings to and from the problem server the initial ACK for the connection is completed but then only the "Client Hello" part of the SSL handshake completes i.e. from packet # 5 the ACK is not received back and retransmissions are continued until the initiating side gives up.
Therefore it seems that because of the way secure SSL handshake works on port 8045 i.e. multiple send/receive of packets to complete the SSL handshake, there is some problem in the network preventing this process. The basic non-secure connection on 8044 which does not use that handshake mechanism does not have a problem.
In summary this is not a XCOM related problem and network team need to investigate further to resolve it.