VMware HCX officially supports RSA key sizes from 2048-bit to 8192-bit for vCenter Machine SSL certificates, including the 3072-bit default used in vSphere 8.x.
Follow these steps to re-establish trust after replacing a vCenter certificate:
Log into the HCX Appliance Management UI by navigating to https://<HCX-Manager-IP-or-FQDN>:9443 and authenticate as the admin user.
Navigate to the vCenter Server configuration section and click Edit. (Credential modifications are not necessary if the service account details remain unchanged.)
Proceed with the configuration wizard. The HCX Manager will reach out to the vCenter Server, detect the new RSA 3072-bit Machine SSL certificate, and prompt with a certificate warning.
Confirm and import the new certificate.
Allow the HCX services (including the app-engine and vSphere Web Client service) to automatically restart to apply the new keystore parameters.
Log into the standard HCX UI at https://<HCX-Manager-IP-or-FQDN>:443.
Navigate to Infrastructure > Site Pairs and edit the site pairing configuration to accept the new remote certificate.
Ensure the applied certificate strictly adheres to standard VMware constraints:
Subject Alternative Name (SAN) must contain DNS Name=<vCenter_FQDN>.
Key Usage / EKU must be correctly defined for Server Authentication.
The signature algorithm must be SHA-256 or higher (SHA1 and MD5 are deprecated).
The complete certificate chain (Leaf/Server -> Intermediate -> Root) must be imported correctly if using a custom CA.