I'm configuring XCOM, to use ssl, and got the following error:
XCOMU0780E Txpi 316: TxpiSSL peer certificate: Certificate verification failed (DNS name does not match) error 5:
The cause of the problem is that they have the following in the configssl.cnf:
# Mandatory, YES/NO
[VERIFY_MACHINE]
INITIATE_SIDE = YES
RECEIVE_SIDE = YES
but you don't have the actual DNS or HOSTNAME defined in the certificates or configssl.cnf.
Release : 11.6
Component : CA XCOM Data Transport for Sun Solaris Operating System
When you want to do "machine verification" with SSL as you have specified, you must generate the certificates with a valid DNS and specify that in the configssl.cnf as the HOST_NAME. No way around that.
If using the CA XCOM sample certificates:
1. remove your existing certificates
2. modify the cassl.conf. You would have to modify 2 sections in the file. Those sections are:
[ SSL_client_extensions ] and [ SSL_server_extensions ]
In those sections you would modify the "subjectAltName = email:copy,DNS:hostname.ca.com", specifically the DNS part. You must put the valid actual DNS of both the client and server. So, it would look like this as an example:
[ SSL_client_extensions ]
subjectAltName = email:copy,DNS:systemA.xxx.com
[ SSL_server_extensions ]
subjectAltName = email:copy,DNS:systemB.xxx.com
Note: I made up the DNS names, you have to specify actual names.
If you don't want "machine" verification, then turn off the parameter in the configssl.cnf.