ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

After upgrade CA XCOM server OS from RHEL 6.9 to 7.7, secure socket ping fails with XCOMU0298E and XCOMU0780E handshake failure

book

Article ID: 186759

calendar_today

Updated On:

Products

XCOM Data Transport XCOM Data Transport - Linux PC

Issue/Introduction

Linux servers that were previously running Linux 6.9 (Santiago) were upgraded to RHEL 7.7 (Maipo).
After the upgrade testing from the RHEL 7.7 source XCOM server to a RHEL 6.10 target XCOM server with command "xcomtcp -ping" on secure socket (8045) shows SSL handshake error as follows:

RHEL 7.7 source:
# xcomtcp -ping REMOTE_SYSTEM=xxx PORT=8045 TRNENCRL_CIPHER=ALL SECURE_SOCKET=YES
XCOMU0298E Unable to allocate remote transaction program: Txpi  215: Socket send error return value = 9

RHEL 6.10 target:
The $XCOM_HOME/xcom.log file contains this corresponding error:
XCOMU0780E Txpi  308: TxpiInitSSL Failed msg = <error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
+++ SSL alert number 40> value = 0:

Environment

Release : 11.6 SP01 17021
Component : CA XCOM Data Transport for Linux PC

Resolution

XCOM support not able to recreate "xcomtcp -ping" problem from RHEL 7.7 to RHEL 6.10 and advised to run the "xcomtcp -ping" command with trace options "-t" and "XTRACE=10" as follows:
# xcomtcp -ping -t REMOTE_SYSTEM=xxx PORT=8045 TRNENCRL_CIPHER=ALL SECURE_SOCKET=YES XTRACE=10 > xcom_ping.out 2>&1
The resulting xcom_ping.out file showed these errors:
===
E - catoossl.c(2899): Error 308

E -catoossl.c(2899): Txpi  308: TxpiInitSSL Failed msg = <error:1408D177:SSL routines:ssl3_get_key_exchange:bad dh g value> value = 4294967295
===

Running openssl command "openssl s_client -connect target_server:8045 -showcerts -status" to the target server also showed error "ssl3_get_key_exchange:bad dh g value" and "Cipher is (NONE)"
XCOM Support observed the same result using openssl command from RHEL 7.7 to RHEL 6.10:
"openssl s_client -connect target_server:8045 -showcerts -status -msg"
"-msg" shows each step of the handshake sequence and the "bad dh g value" error occurs after the server key exchange step:
ClientHello
ServerHello
Certificate
ServerKeyExchange

NOTE: XCOM uses its own CAPKI SSL libraries and not the system openssl libraries.

On the target server, which is effectively the SSL server, the configssl.cnf file [RSAKEY] parameter "RECEIVE_SIDE" controls whether RSA or Diffie Hellman (DH) key exchange is used by XCOM as follows:
1. The OOTB default value for [RSAKEY] parameter "RECEIVE_SIDE" is "NO" which means Diffie Hellman key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was DHE-RSA-AES256-SHA. For the reported problems "RECEIVE_SIDE = NO" was being used.
2. When [RSAKEY] parameter "RECEIVE_SIDE" is changed to "YES" it means an RSA key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was AES256-SHA. After the option "RECEIVE_SIDE = YES" was implemented it resolved both the "xcomtcp -ping" and openssl errors. Therefore it appears that the RHEL 7.7 upgrade had somehow introduced a problem which prevented use of the Diffie Hellman key exchange that did not exist before. Additional investigation using [RSAKEY] parameter "RECEIVE_SIDE = NO" found that DH could be used successfully for both "xcomtcp -ping" and openssl tests if a prime of 1024 instead of the default 256 was used 

So the 2 possible resolutions for target server config.ssl.cnf file are:
1. Set [RSAKEY] parameter RECEIVE_SIDE to YES so that RSA key exchange is used i.e.
===
[RSAKEY]
RECEIVE_SIDE = YES
===

2. Set [RSAKEY] parameter RECEIVE_SIDE to NO so that DH key exchange is used and also use a DH prime of 1024 i.e.
===
[RSAKEY]

RECEIVE_SIDE = NO

# Optional (for RSA NO, see above). If RSA NO and DH files empty,
#  then internal program tables will be used.
[DH]
# DH_512_RECEIVE_SIDE  = /opt/CA/XCOM/config/dh512.pem
DH_1024_RECEIVE_SIDE = /opt/CA/XCOM/config/dh1024.pem
# DH_2048_RECEIVE_SIDE = /opt/CA/XCOM/config/dh2048.pem
# DH_4096_RECEIVE_SIDE = /opt/CA/XCOM/config/dh4096.pem
===

However as RSA is more secure than DH it is normally advised to use the RSA option.

Additional Information

CA XCOM SSL - How to set up RSA or Diffie Hellman (DH) for key exchange in file configssl.cnf