When using certificate authority (CA) there are specific requirements.
When self signed certificates are replace with customer CA certificate if they do not meet specific requirements communications will fail.
Requirements for All Imported vSphere Certificates
Key size: 2048 bits (minimum) to 8192 bits (maximum) (PEM encoded). The vSphere Client and API still accept a key size up to 16384 bits when generating the Certificate Signing Request.
In vSphere 8.0, you can only generate CSRs with a minimum key length of 3072 bits when using the vSphere Client or the vSphere Certificate Manager. vCenter Server still does accept custom certificates bearing a key length of 2048 bits.
n vSphere 8.0 Update 1 and later, you can use the vSphere Client to generate a CSR with a key length of 2048 bits.
vSphere's FIPS certificate only validates RSA key sizes of 2048 bits and 3072 bits.
PEM format. VMware supports PKCS8 and PKCS1 (RSA keys). When you add keys to VECS, they are converted to PKCS8.
x509 version 3
SubjectAltName must contain DNS Name=machine_FQDN
CRT format
Exempting the vpxd-extension solution user certificate, Extended Key Usage can be either empty or contain Server Authentication.
vSphere does not support the following certificates.
Certificates with wildcards.
The algorithms md2WithRSAEncryption, md5WithRSAEncryption, RSASSA-PSS, dsaWithSHA1, ecdsa_with_SHA1, and sha1WithRSAEncryption are not supported.
When creating a custom machine SSL certificate for vCenter Server, Server Authentication and Client Authentication are not supported, and must be removed when using the Microsoft Certificate Authority (CA) templates.
Encrypted private keys are not supported.
The error message: " Http failure response for https://<IP or FQDN >/api/admin/certificates/serverCertificate/httpd: 500 OK" and "Error, unable to connect via certificate provided."
This is an example of the error when an encrypted private key is attempted in the HCX product.
When viewing the encrypted key it will state that it is encrypted. The supported key will not have "ENCRYPTED" in the BEGIN and END tags.
Environment
VMware HCX
Cause
Encrypted Private Keys cannot be used in the "update certificate" workflow for HCX Manager.
Resolution
Use a custom CA that does not have an encrypted private key.