'Hostname mismatch' or 'server certificate could not be verified' error occurs when attempting a cold migration in VMware Cloud Director Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

• When performing a cold migration with VMware Cloud Director Availability 4.x, an error similar to the following is encountered on the source machine:
  
  hostname mismatch
  
  or
  
  The server certificate could not be verified
  
  
• The primary certificate on the destination replicator is valid and was signed by a third party certificate authority.
• /var/log/h4dm-agent.log on failed VM/Workstation revealed expired h4dm certificates
  
  [YYYY-MM-DDTHH:MM:SS.#######[0m [34mDEBUG[0m [2mepic::reqwest_errors[0m[2m:[0m native-tls error: Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "../ssl/statem/statem_clnt.c", line: 1889 }]))) }, X509VerifyResult { code: 10, error: "certificate has expired" })
  [YYYY-MM-DDTHH:MM:SS.#######[0m [34mDEBUG[0m [2mepic[0m[2m:[0m code: TlsInvalidCert, args: [], msg: The server certificate could not be verified., orig_msg: error sending request for url (https://dm-######-###-###-####-########/os-based/cold-migrations): error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (certificate has expired)
  
  
• The certificate returned on port 3030 of the destination replicator appliance has expired or is no longer valid

Environment

VMware Cloud Director Availability 4.x

Cause

Cold migration and other limited use cases will replicate traffic to the data mover service that listens to port 3030 on the destination replicator, which utilizes a separate certificate than the item installed via the replicator appliance user interface. This certificate is updated when the primary user interface certificate is regenerated locally, but will not be altered if an externally signed certificate is uploaded. Thus, if external certificates are used, this data mover certificate can become out of date/expired since it is unaltered.

Resolution

To resolve this issue, we need to renew the replicator/h4dm certificates. 

1. If the replicator certificates are expired, regenerate the certificate using the following instructions: Replace the SSL certificate of the Replicator Service
2. If replicator was configured CA certificates, re-apply the original CA certificate, following the documented instructions above.
3. To regenerate only the h4dm certificates on replicator appliance, perform these actions:
   
   a. Rename '/opt/vmware/h4/h4dm/conf/cert.pem' and '/opt/vmware/h4/h4dm/conf/key.pem'    (for ex: cert.pem.bak and key.pem.bak)
   b. Restart replicator appliance
   
   
4. Re-issue the cold migration ISO.