XCOM support not able to recreate "xcomtcp -ping" problem from RHEL 7.7 to RHEL 6.10 and advised to run the "xcomtcp -ping" command with trace options "-t" and "XTRACE=10" as follows:
# xcomtcp -ping -t REMOTE_SYSTEM=xxx PORT=8045 TRNENCRL_CIPHER=ALL SECURE_SOCKET=YES XTRACE=10 > xcom_ping.out 2>&1
The resulting xcom_ping.out file showed these errors:
E - catoossl.c(2899): Error 308
E -catoossl.c(2899): Txpi 308: TxpiInitSSL Failed msg = <error:1408D177:SSL routines:ssl3_get_key_exchange:bad dh g value> value = 4294967295
Running openssl command "openssl s_client -connect target_server:8045 -showcerts -status" to the target server also showed error "ssl3_get_key_exchange:bad dh g value" and "Cipher is (NONE)"
XCOM Support observed the same result using openssl command from RHEL 7.7 to RHEL 6.10:
"openssl s_client -connect target_server:8045 -showcerts -status -msg"
"-msg" shows each step of the handshake sequence and the "bad dh g value" error occurs after the server key exchange step:
NOTE: XCOM uses its own CAPKI SSL libraries and not the system openssl libraries.
On the target server, which is effectively the SSL server, the configssl.cnf file [RSAKEY] parameter "RECEIVE_SIDE" controls whether RSA or Diffie Hellman (DH) key exchange is used by XCOM as follows:
1. The OOTB default value for [RSAKEY] parameter "RECEIVE_SIDE" is "NO" which means Diffie Hellman key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was DHE-RSA-AES256-SHA. For the reported problems "RECEIVE_SIDE = NO" was being used.
2. When [RSAKEY] parameter "RECEIVE_SIDE" is changed to "YES" it means an RSA key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was AES256-SHA. After the option "RECEIVE_SIDE = YES" was implemented it resolved both the "xcomtcp -ping" and openssl errors. Therefore it appears that the RHEL 7.7 upgrade had somehow introduced a problem which prevented use of the Diffie Hellman key exchange that did not exist before. Additional investigation using [RSAKEY] parameter "RECEIVE_SIDE = NO" found that DH could be used successfully for both "xcomtcp -ping" and openssl tests if a prime of 1024 instead of the default 256 was used
So the 2 possible resolutions for target server config.ssl.cnf file are:
1. Set [RSAKEY] parameter RECEIVE_SIDE to YES so that RSA key exchange is used i.e.
RECEIVE_SIDE = YES
2. Set [RSAKEY] parameter RECEIVE_SIDE to NO so that DH key exchange is used and also use a DH prime of 1024 i.e.
RECEIVE_SIDE = NO
# Optional (for RSA NO, see above). If RSA NO and DH files empty,
# then internal program tables will be used.
# DH_512_RECEIVE_SIDE = /opt/CA/XCOM/config/dh512.pem
DH_1024_RECEIVE_SIDE = /opt/CA/XCOM/config/dh1024.pem
# DH_2048_RECEIVE_SIDE = /opt/CA/XCOM/config/dh2048.pem
# DH_4096_RECEIVE_SIDE = /opt/CA/XCOM/config/dh4096.pem
However, as RSA is more secure than DH it is normally advised to use the RSA option.
Related article: XCOM configssl.cnf key exchange for RSA or Diffie Hellman (DH)