HCX - Mobility Agent (MA) deployment fails when vCenter 'vpxd.certmgmt.mode' is set to 'custom' mode

Products

VMware HCX

Issue/Introduction

HCX Service Mesh configuration workflow fails deploy Mobility Agent (MA) virtual host with error message:

applianceLifecycle job failed | interconnectConfigMA failed |error Adding Mobility Agent Host failed | SSL Exception

From the vCenter Monitor Tasks, the HCX attempt to deploy MA host fails at ~80%.
This issue will not impact the deployment of HCX Interconnect (IX) and Network Extension (NE) appliances and tunnel status will be up.

vCenter vpxd logs (/var/log/vmware/vpxd/vpxd.log) will show the following error message:

ERROR c.v.v.h.s.i.InterconnectConfigureMA- Task task-####error out, error : A general system error occurred: SSL  Exception: Verification parameters:
PeerThumbprint: ##:##:##:##:##:##
ExpectedThumbprint:
ExpectedPeerName: #.#.#.#
The remote host certificate has these problems:
 * Host name does not match the subject name(s) in certificate.
 * unable to get local issuer certificate
com.vmware.vim.binding.vmodl.fault.SystemError: A general system error occurred: SSL Exception: Verification parameters:
PeerThumbprint: ##:##:##:##:##:##
ExpectedThumbprint:
ExpectedPeerName: #.#.#.#
The remote host certificate has these problems:
* Host name does not match the subject name(s) in certificate.

Environment

VMware HCX
vCenter Server

Cause

The issue will occur if vCenter is configured with Certificate Management policy set to "custom" mode under advanced parameter "vpxd.certmgmt.mode"
HCX Mobility Agent (MA) virtual host must be added by HCX Manager into vCenter but since the IX appliance uses self-signed certificate, vCenter will reject the addition of the MA into the cluster.

Resolution

HCX MA deployment in a vCenter environment with Certificate Management set to "custom" mode is NOT supported.
Alternatively the recommended workaround can be implemented and that has been thoroughly verified, yet it has some restrictions to be persistent.
Support for the the implementation of the workaround is provided as a best effort.

Workaround:

The following procedure will replace the IX appliance certificate and key.
It will have to be performed for each Interconnect appliance that is deployed in a vCenter with 'custom' certificate management.

SSH into the HCX Manager as 'admin'
Enable ccli mode
list the appliances and go <appliance_ID> into the IX appliance

ssh to drop into the linux prompt

Example from a lab for reference:

admin@hcx [ ~ ]$ ccli
Welcome to HCX Central CLI
[admin@hcx] list
|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| Node                     	| Id | Address              | State     | HcxManagerCert                     | HcxManagerKey                      | Selected |
|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| SM-IX-I1  				| 0  | <IX-IP>:9443 		| Connected | /common/cs/hm_to_gw_1730796807.pem | /common/ks/hm_to_gw_1730796807.pk8 |          |

[admin@hcx] go 0
Switched to node 0.
[admin@hcx:SM-IX-I1] ssh

Change directory to /etc/vmware/ssl

Backup certificate files:

mv rui.crt rui.crt.bak
mv rui.key rui.key.bak

Replace the files with the custom CA cert and key
Reboot the IX appliance or restart the MA and authentication services :
```
stc restart mobilityagent
stc restart authdlauncher
```

From the HCX Interconnect UI, re-sync the Service Mesh to trigger the MA deployment.

IMPORTANT: This workaround will not be persistent if the Service Mesh is re-synced or after service updates. The same procedure will have to be performed to re-deploy the MA again.

The following considerations should be taken into account:

As this is a custom CA cert to be trusted by vCenter, it must be generated in the same manner that the existing ESXi host certificates have been generated for this environment.
Ensure the 'CN' and 'SAN' fields of the certificate contain the IP address that is intended to be used for the management IP of the IX appliance.
If the certificate generated by the CA is provided as a cert.pem file, make sure the certificate chain (including target, intermediate, and root certificates) and private key aspects are separated into the rui.cert and rui.key files.

Additional Information

Refer to the following KB articles for more information on how to request and configure CA signed certificates for ESXi hosts

KB: Configuring CA signed certificates for ESXi hosts
KB: Configuring OpenSSL for installation and configuration of CA signed certificates in the vSphere environment\

Impact/Risks:
The failure to deploy the MA will disable HCX Cold, vMotion, and RAV migration services.
VR Bulk migrations and DR Protections do not require MA.
There is no risk in implementing the recommended workaround as there is no impact to Network Extension services.