"INVALID_KEY, the private key does not match the certificate" while replacing the machine SSL certificate or the VMCA root certificate against a custom CA certificate

Products

• VMware vCenter Server

Issue/Introduction

• Using the vSphere Certificate Manager CLI to replace the Machine SSL or Root Certificate with a Custom Certificate fails. It shows the following error during replacement:

ERROR:: INVALID_KEY, the private key does not match the certificate. Please provide a valid certificate and Key pair.

• Certificate-manager CLI utility is used to generate the CSR needed to replace the VC machine SSL certificate or the VMCA root certificate and the output dir is provided as /tmp.

Environment

• vCenter Server 8.0 U2 and above.

Cause

Despite being the super user, the root user is unable to access the files in /tmp which are owned by "vmcad-user" and "lwis" group due to security restrictions, this results in a mismatch in the private and csr pair.

Resolution

This is a known issue affecting vCenter Server  version from 8.0 Update 2 until including 8.0 Update 3j.

A fix has been released with vCenter Server 8.0 Update 3i (build 25205845), available via the Broadcom download portal.

If you currently unable to update to this or a later version, work around this by using different export location than /tmp for the CSR and the key. For example you can use /storage/core as the target location for both the certificate request and key.

Additional Information

• Use the following commands to run an MD5 check for CSR, Private Key and the Certificate. The values should match if the Certificate is signed using the same Private Key and CSR pair:

1. 1. To view the md5 hash of the modulus of the CSR:

      # openssl req -in <path to the CSR file> -noout -modulus | openssl sha256

   2. To view the md5 hash of the modulus of the Private Key:

      # openssl rsa -in <path to the Private Key file> -noout -modulus | openssl sha256

   3. To view the md5 hash of the modulus of the certificate:

      # openssl x509 -in <path to the Certificate file> -noout -modulus | openssl sha256

Example: 

openssl rsa -noout -modulus -in vmca_issued_key.key |openssl sha256 
SHA2-256(stdin)= 1111#######################################################1111

openssl req -noout -modulus -in vmca_issued_csr.csr |openssl sha256 
SHA2-256(stdin)= 1111#######################################################1111

Further references:

• Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate.
• Error: "The provided MACHINE_SSL certificate and provided private key are not valid" when performing SSL certificate renewal on vCenter.