ESXi Custom Certificate Import Fails with "The generated private key does not match with the public key from the incoming certificate" error in vCenter vSphere Client

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• When attempting to import a custom certificate to an ESXi host through the vCenter vSphere Client, the process fails with the following error message:
  
  • "The generated private key does not match with the public key from the incoming certificate"
    
    

Environment

VMware vSphere ESXi 8.x

Cause

This error stems from a cryptographic mismatch where the private key (rui.key) and the certificate file (rui.crt) fail to pair. A comparison of their SHA-256 thumbprints confirms that the existing private key does not correspond to the provided certificate.

Resolution

1. Establish an SSH connection to the ESXi host using root user. 
2. Create a configuration file as outlined in the article : Configuring OpenSSL for installation and configuration of CA signed certificates in the vSphere environment
3. Run the below command to generate the Certificate Signing Request (CSR):
   • openssl req -new -nodes -out filename.csr -newkey rsa:2048 -keyout filename.key -config filename.cfg 
4. Submit the Certificate Signing Request(CSR) to a Certificate Authority (CA) for signing.
5. Use OpenSSL to compare the SHA-256 thumbprints of the private and public keys, and verify that they match.
   
   • openssl pkey -in rui.key -pubout -outform pem | sha256sum
   • openssl x509 -in rui.crt -pubkey -noout -outform pem | sha256sum
6. Create a backup of the castore.pem file using the command below.
   
   • cp /etc/vmware/ssl/castore.pem /etc/vmware/ssl/castore.pem.bak

7. Copy the signing certificate to /etc/vmware/ssl/Root.cer

   Note: If the user has one or more intermediate certificate authorities, the signing certificate must be a chain of all intermediate certificates and the root certificate. 

8. Append the root certificate to the castore.pem file using the following command:

   • cat Root.cer >> castore.pem

9. Back up the old .crt and .key files by renaming them using the commands below:

   • mv rui.crt orig.rui.crt
   • mv rui.key orig.rui.key
10. Copy the custom certificate chain (rui.crt) and private key (rui.key) files to the /etc/vmware/ssl directory.
11. Restart the ESXi host services for the changes to take effect.
    
    • /etc/init.d/services.sh restart