vCenter Server fails to replace MACHINE_SSL custom certificate signed with Signature algorithm RSASSA-PSS

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
Unable to connect to vCenter Server after attempting to replace the MACHINE_SSL with a custom certificate.

• vmware-vpxd service fails to start

Logs reflect an error with the certificate:
 

/var/log/vmware/vmware-vpxd/vpxd.log

• "failed to read x509 cert; err: 151441516"
• "sub=IO.Connection Failed to connect; e: 111 (Connection refused)"
• "sub=HttpConnectionPool-000071 Failed to get pooled connection; Connection Refused"

/var/log/vmware/lookupsvc/lookupserver-default.log

[2023-08-03T22:01:30.139Z pool-2-thread-5 ERROR com.vmware.vim.lookup.vlsi.util.VmodlEnhancer] java x.net.ssl.SSLException: SSL handshake from 0.0.0.0/0.0.0.0:XXXXX to vcsa/127.0.0.1:443 failed in 8 ms

com.vmware.vim.sso.admin.exception.InternalError: javax.net.ssl.SSLException: SSL handshake from 0.0.0.0/0.0.0.0:XXXXX to vcsa/127.0.0.1:443 failed in 8 ms

Caused by: org.bouncycastle.tls.TlsFatalAlert: bad_certificate(42)
        at org.bouncycastle.tls.TlsUtils.checkSigAlgOfServerCerts(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The vmware-vpxd service fails to start due to the lookupsvc failing to load the machine_ssl certificate. The custom machine_ssl certificate, has been signed with unsupported PKCS #1 v2.1 Signature algorithm RSASSA-PSS.

The custom machine_ssl certificate must use the following Signatures:

• Signature algorithm sha256RSA
• Signature hash algorithm sha256

Resolution

Contact the Public Key Infrastructure (PKI) team that issued the custom machine_ssl certificate to update the Certificate Authority to renew/issue the certificate using Signature algorithm sha256RSA and Signature hash algorithm sha256.



Note, If validation fails on the certificates, it may also fail the PKI chain if also signed using Signature algorithm RSASSA-PSS. Certificate Authorities in the chain will need to be renewed using Signature algorithm sha256RSA.

Additional Information

"ERROR certificate-manager 'lstool get-site-id' failed: 1", Certificate Replacement with Custom Certificate Fails on vCenter Server 6.x (71120)
https://knowledge.broadcom.com/external/article/344262 

Replace Machine SSL Certificates with Custom Certificates
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/replace-machine-ssl-certificate-with-custom-certificate.html

Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.x/7.x (2112009)
https://knowledge.broadcom.com/external/article/315271 

vmware-vpxd service not starting (83113)
https://knowledge.broadcom.com/external/article/322158