After having configured HSM integration between ProxySG (SG) and Luna SP with mutual authentication, it is necessary to test mutual authentication with the following command: test hsm-keyring <keyring-name>
If all is set up correctly and working, you should receive a message indicating success. However, there are a number of reasons you might receive an error message. This article is specific to the "Hostname in server certificate does not match Hostname" error message. This message indicates that the host that the SG was expecting was not the same as what was presented in the common-name subject field on the certificate returned by the Luna SP appliance. This validation (Verify Peer in device profile) is performed for security reasons and is enable by default in order to avoid connecting to the wrong host.
In SGOS versions 184.108.40.206 to 220.127.116.11, there is a limitation in the HSM configuration settings which accepts only an IP address as the HSM host. Therefore, if you have a hostname as the common-name of the certificate of the Luna SP appliance, it will fail validation because the SG is expecting the IP address only since that is all that can currently be entered in the settings.
The setting is found in the Management Console under Configuration>SSL>HSM:
As of the date of this article, Blue Coat is in the process of implementing an enhancement to allow a hostname in the HSM settings so that a common-name containing a hostname will be supported. This enhancement will be released in a post-18.104.22.168 version, with an ETA of about June-July 2015.
Aside from creating a certificate on the Luna SP appliance that has the IP address as the common-name instead of it's hostname, a workaround would be to disable the Verify Peer setting in the device profile used for HSM on the SG. However, keep in mind that while simply disabling this option might be an easy workaround, mutual authentication will not actually take place since only the Luna SP will be validating the certificate from the SG and not vice-versa.