I'm configuring XCOM, to use ssl, and got the following error:
XCOMU0780E Txpi 316: TxpiSSL peer certificate: Certificate verification failed (DNS name does not match) error 5:
Release : 11.6
Component : CA XCOM Data Transport for Sun Solaris Operating System
The cause of the problem is that they have the following in the configssl.cnf:
# Mandatory, YES/NO
INITIATE_SIDE = YES
RECEIVE_SIDE = YES
but you don't have the actual DNS or HOSTNAME defined in the certificates or configssl.cnf.
When you want to do "machine verification" with SSL as you have specified, you must generate the certificates with a valid DNS and specify that in the configssl.cnf as the HOST_NAME. No way around that.
If using the CA XCOM sample certificates:
1. remove your existing certificates
2. modify the cassl.conf. You would have to modify 2 sections in the file. Those sections are:
[ SSL_client_extensions ] and [ SSL_server_extensions ]
In those sections you would modify the "subjectAltName = email:copy,DNS:hostname.ca.com", specifically the DNS part. You must put the valid actual DNS of both the client and server. So, it would look like this as an example:
[ SSL_client_extensions ]
subjectAltName = email:copy,DNS:systemA.xxx.com
[ SSL_server_extensions ]
subjectAltName = email:copy,DNS:systemB.xxx.com
Note: I made up the DNS names, you have to specify actual names.
If you don't want "machine" verification, then turn off the parameter in the configssl.cnf.