HCX Mobility Agent (MA) deployment fails when vCenter 'vpxd.certmgmt.mode' is set to 'custom' mode
search cancel

HCX Mobility Agent (MA) deployment fails when vCenter 'vpxd.certmgmt.mode' is set to 'custom' mode

book

Article ID: 316379

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

  • HCX Service Mesh configuration workflow fails deploy Mobility Agent (MA) virtual host with error message:
     
    applianceLifecycle job failed | interconnectConfigMA failed |error Adding Mobility Agent Host failed | SSL Exception

     

    OR
     
    Service Mesh modification failed. Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. A general system error occurred: Failed to verify certificate on <MA IP>. When ESXi Certificate Mode is set to custom it is mandatory to install valid certificate on ESXi host before adding the host to a VC

  • From the vCenter Monitor Tasks, the HCX attempt to deploy MA host fails at ~80%.

  • This issue will not impact the deployment of HCX Interconnect (IX) and Network Extension (NE) appliances and tunnel status will be up.

  • vCenter vpxd logs (/var/log/vmware/vpxd/vpxd.log) will show the following error message: 

    ERROR c.v.v.h.s.i.InterconnectConfigureMA- Task task-####error out, error : A general system error occurred: SSL  Exception: Verification parameters:
PeerThumbprint: ##:##:##:##:##:##
ExpectedThumbprint:
ExpectedPeerName: #.#.#.#
The remote host certificate has these problems:
 * Host name does not match the subject name(s) in certificate.
 * unable to get local issuer certificate
com.vmware.vim.binding.vmodl.fault.SystemError: A general system error occurred: SSL Exception: Verification parameters:
PeerThumbprint: ##:##:##:##:##:##
ExpectedThumbprint:
ExpectedPeerName: #.#.#.#
The remote host certificate has these problems:
* Host name does not match the subject name(s) in certificate.

Environment

VMware HCX
vCenter Server

Cause

The issue will occur if vCenter is configured with Certificate Management policy set to "custom" mode under advanced parameter "vpxd.certmgmt.mode" 
HCX Mobility Agent (MA) virtual host must be added by HCX Manager into vCenter but since the IX appliance uses self-signed certificate, vCenter will reject the addition of the MA into the cluster.

Resolution

HCX MA deployment in a vCenter environment with Certificate Management set to "custom" mode is NOT supported.
Alternatively the recommended workaround can be implemented and that has been thoroughly verified, yet it has some restrictions to be persistent.
Support for the the implementation of the workaround is provided as a best effort.

Workaround:

The following procedure will replace the IX appliance certificate and key.
It will have to be performed for each Interconnect appliance that is deployed in a vCenter with 'custom' certificate management.

  • SSH into the HCX Manager as 'admin'
  • Enable ccli mode
  • list the appliances and go <appliance_ID> into the IX appliance
  • ssh to drop into the linux prompt

    Example from a lab for reference: 

    admin@hcx [ ~ ]$ ccli
Welcome to HCX Central CLI
[admin@hcx] list
|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| Node                     	| Id | Address              | State     | HcxManagerCert                     | HcxManagerKey                      | Selected |
|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| SM-IX-I1  				| 0  | <IX-IP>:9443 		| Connected | /common/cs/hm_to_gw_1730796807.pem | /common/ks/hm_to_gw_1730796807.pk8 |          |

[admin@hcx] go 0
Switched to node 0.
[admin@hcx:SM-IX-I1] ssh


  • Change directory to /etc/vmware/ssl
  • Backup certificate files: 
    mv rui.crt rui.crt.bak
mv rui.key rui.key.bak
  • ​​Replace the files with the custom CA cert and key
  • Reboot the IX appliance or restart the MA and authentication services :

    systemctl restart mobilityagent
systemctl restart authdlauncher
  • From the HCX Interconnect UI, re-sync the Service Mesh to trigger the MA deployment. 

IMPORTANT: This workaround will not be persistent if the Service Mesh is re-synced or after service updates. The same procedure will have to be performed to re-deploy the MA again.

The following considerations should be taken into account:

  • As this is a custom CA cert to be trusted by vCenter, it must be generated in the same manner that the existing ESXi host certificates have been generated for this environment.
  • Ensure the 'CN' and 'SAN' fields of the certificate contain the IP address that is intended to be used for the management IP of the IX appliance.
  • If the certificate generated by the CA is provided as a cert.pem file, make sure the certificate chain (including target, intermediate, and root certificates) and private key aspects are separated into the rui.cert and rui.key files.

IMPORTANT:

    • In certain scenarios the cert constructed will grow larger than the envoy service can consume (4096 bytes).
    • To work around this, instead of adding the intermediate/leaf and root certificate to the rui.crt file, add it to the castore.pem file.
      • The castore.pem file can be found in /etc/vmware/ssl folder on the IX appliance.
      • Simply "vi" the file and add the required certs (intermediate/leaf and root certificates).

 



Additional Information

Refer to the following KB articles for more information on how to request and configure CA signed certificates for ESXi hosts 

Powered by Wolken Software