Error: "INVALID_KEY, the private key does not match the certificate" received when trying to replace the Machine SSL or Root Certificate with a Custom Certificate

Products

• VMware vCenter Server

Issue/Introduction

• Using the vSphere Certificate Manager CLI to replace the Machine SSL or Root Certificate with a Custom Certificate fails. It shows the following error during replacement:

ERROR:: INVALID_KEY, the private key does not match the certificate. Please provide a valid certificate and Key pair.

• Certificate-manager CLI utility is used to generate the CSR needed to replace the VC machine SSL certificate or the VMCA root certificate and the output dir is provided as /tmp.

Environment

• vCenter Server 8.0 U2 and above.

Cause

Despite being the super user, the root user is unable to access the files in /tmp which are owned by "vmcad-user" and "lwis" group due to security restrictions, this results in a mismatch in the private and csr pair.

Resolution

This is a known issue with vCenter Server 8.0 U2 and above versions. Broadcom Engineering is actively working towards fixing this issue in a future patch release.

Workaround

• Do not use /tmp as a directory location to export the CSR and the key, but instead use a different folder (such as /storage/core) as the target location for both the certificate request and key.

Additional Information

• Use the following commands to run an MD5 check for CSR, Private Key and the Certificate. The values should match if the Certificate is signed using the same Private Key and CSR pair:

1. 1. To view the md5 hash of the modulus of the CSR:

      # openssl req -in <path to the CSR file> -noout -modulus | openssl sha256

   2. To view the md5 hash of the modulus of the Private Key:

      # openssl rsa -in <path to the Private Key file> -noout -modulus | openssl sha256

   3. To view the md5 hash of the modulus of the certificate:

      # openssl x509 -in <path to the Certificate file> -noout -modulus | openssl sha256

Example: 

openssl rsa -noout -modulus -in vmca_issued_key.key |openssl sha256 
SHA2-256(stdin)= 1111#######################################################1111

openssl req -noout -modulus -in vmca_issued_csr.csr |openssl sha256 
SHA2-256(stdin)= 1111#######################################################1111

Further references:

• Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate.
• Error: "The provided MACHINE_SSL certificate and provided private key are not valid" when performing SSL certificate renewal on vCenter.