XCOM for Windows 11.6 SP02 server newly configured for SSL transfers fails with error:
#XCOMN0298E Unable to allocate remote transaction program: XCOMN0780E Txpi 308: TxpiInitSSL Failed msg = <error:0906D06C:PEM routines:PEM_read_bio:no start line +++ Expecting: TRUSTED CERTIFICATE> value = 0:
The error means that the initial SSL handshake is failing and therefore the transfer itself is not being executed.
On the XCOM server which is generating the error message first confirm if the same/similar error occurs for an SSL loopback transfer i.e. execute a transfer test on that server with "REMOTE_SYSTEM=127.0.0.1". If the error is repeated then it indicates an SSL configuration problem on that XCOM server.
The XCOM SSL configuration requires the use of PEM encoded certificate/private key files.
The error message points to one of those files having a problem e.g. the file may be corrupt or not have the expected PEM encoding (perhaps DER encoded instead), or a private key file is being found where a certificate is file is expected.
Check the %XCOM_HOME%\config\configssl.cnf file and in particular, verify that the [CERTIFICATE] and [PRIVATEKEY] sections are pointing to the correct file names and that those files have the expected contents e.g. the default configssl.cnf contains:
=====
...
# Mandatory
[CA]
INITIATE_SIDE = %XCOM_HOME%\\ssl\\certs\\cassl.pem
RECEIVE_SIDE = %XCOM_HOME%\\ssl\\certs\\cassl.pem
# Mandatory
[CA_DIRECTORY]
INITIATE_SIDE = %XCOM_HOME%\\ssl\\certs
RECEIVE_SIDE = %XCOM_HOME%\\ssl\\certs
# Mandatory
[CERTIFICATE]
INITIATE_SIDE = %XCOM_HOME%\\ssl\\certs\\clientcert.pem
RECEIVE_SIDE = %XCOM_HOME%\\ssl\\certs\\servercert.pem
# Mandatory
[PRIVATEKEY]
INITIATE_SIDE = %XCOM_HOME%\\ssl\\private\\clientkey.pem
RECEIVE_SIDE = %XCOM_HOME%\\ssl\\private\\serverkey.pem
...
=====
XCOM Data Transport for Windows 12.0 Generate TLS/SSL Certificates
Using XCOM with CA (Certificate Authority) SSL certificates
OpenSSL for Windows can be installed (OpenSSL Binaries) and then used to check the file contents e.g.
"C:\Program Files\OpenSSL-Win64\bin\openssl" x509 -in "%XCOM_HOME%\ssl\certs\clientcert.pem" -text -noout > clientcert.txt
"C:\Program Files\OpenSSL-Win64\bin\openssl" x509 -in "%XCOM_HOME%\ssl\certs\servercert.pem" -text -noout > servercert.txt
"C:\Program Files\OpenSSL-Win64\bin\openssl" rsa -in "%XCOM_HOME%\ssl\private\clientkey.pem" -text -noout > clientkey.txt
"C:\Program Files\OpenSSL-Win64\bin\openssl" rsa -in "%XCOM_HOME%\ssl\private\serverkey.pem" -text -noout > serverkey.txt