After upgrade CA XCOM server OS from RHEL 6.9 to 7.7, secure socket ping fails with XCOMU0298E and XCOMU0780E handshake failure
Article ID: 186759
CA XCOM Data TransportCA XCOM Data Transport - Linux PC
Linux servers that were previously running Linux 6.9 (Santiago) were upgraded to RHEL 7.7 (Maipo). After the upgrade testing from the RHEL 7.7 source XCOM server to a RHEL 6.10 target XCOM server with command "xcomtcp -ping" on secure socket (8045) shows SSL handshake error as follows:
RHEL 6.10 target: The $XCOM_HOME/xcom.log file contains this corresponding error: XCOMU0780E Txpi 308: TxpiInitSSL Failed msg = <error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure +++ SSL alert number 40> value = 0:
Release : 11.6 SP01 17021 Component : CA XCOM Data Transport for Linux PC
XCOM support not able to recreate "xcomtcp -ping" problem from RHEL 7.7 to RHEL 6.10 and advised to run the "xcomtcp -ping" command with trace options "-t" and "XTRACE=10" as follows: # xcomtcp -ping -t REMOTE_SYSTEM=xxx PORT=8045 TRNENCRL_CIPHER=ALL SECURE_SOCKET=YES XTRACE=10 > xcom_ping.out 2>&1 The resulting xcom_ping.out file showed these errors: === E - catoossl.c(2899): Error 308
E -catoossl.c(2899): Txpi 308: TxpiInitSSL Failed msg = <error:1408D177:SSL routines:ssl3_get_key_exchange:bad dh g value> value = 4294967295 ===
Running openssl command "openssl s_client -connect target_server:8045 -showcerts -status" to the target server also showed error "ssl3_get_key_exchange:bad dh g value" and "Cipher is (NONE)" XCOM Support observed the same result using openssl command from RHEL 7.7 to RHEL 6.10: "openssl s_client -connect target_server:8045 -showcerts -status -msg" "-msg" shows each step of the handshake sequence and the "bad dh g value" error occurs after the server key exchange step: ClientHello ServerHello Certificate ServerKeyExchange
NOTE: XCOM uses its own CAPKI SSL libraries and not the system openssl libraries.
On the target server, which is effectively the SSL server, the configssl.cnf file [RSAKEY] parameter "RECEIVE_SIDE" controls whether RSA or Diffie Hellman (DH) key exchange is used by XCOM as follows: 1. The OOTB default value for [RSAKEY] parameter "RECEIVE_SIDE" is "NO" which means Diffie Hellman key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was DHE-RSA-AES256-SHA. For the reported problems "RECEIVE_SIDE = NO" was being used. 2. When [RSAKEY] parameter "RECEIVE_SIDE" is changed to "YES" it means an RSA key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was AES256-SHA. After the option "RECEIVE_SIDE = YES" was implemented it resolved both the "xcomtcp -ping" and openssl errors. Therefore it appears that the RHEL 7.7 upgrade had somehow introduced a problem which prevented use of the Diffie Hellman key exchange that did not exist before. Additional investigation using [RSAKEY] parameter "RECEIVE_SIDE = NO" found that DH could be used successfully for both "xcomtcp -ping" and openssl tests if a prime of 1024 instead of the default 256 was used So the 2 possible resolutions for target server config.ssl.cnf file are: 1. Set [RSAKEY] parameter RECEIVE_SIDE to YES so that RSA key exchange is used i.e. === [RSAKEY] RECEIVE_SIDE = YES ===
2. Set [RSAKEY] parameter RECEIVE_SIDE to NO so that DH key exchange is used and also use a DH prime of 1024 i.e. === [RSAKEY] RECEIVE_SIDE = NO
# Optional (for RSA NO, see above). If RSA NO and DH files empty, # then internal program tables will be used. [DH] # DH_512_RECEIVE_SIDE = /opt/CA/XCOM/config/dh512.pem DH_1024_RECEIVE_SIDE = /opt/CA/XCOM/config/dh1024.pem # DH_2048_RECEIVE_SIDE = /opt/CA/XCOM/config/dh2048.pem # DH_4096_RECEIVE_SIDE = /opt/CA/XCOM/config/dh4096.pem ===
However as RSA is more secure than DH it is normally advised to use the RSA option.