Error: "Verification of the machine SSL certificate failed due to an invalid trusted root certificate chain" during vCenter Server upgrade/update

Products

• VMware vCenter Server 8.0

Issue/Introduction

vCenter Server upgrade pre-check shows below error message:

Verification of the machine SSL certificate failed due to an invalid trusted root certificate chain. ERROR: ['unable to get issuer certificate']. Unable to find the root certificate with the subject '<X509Name object '/C=##/ST=#####/L=#####/O=######.###, ###./CN=#######'>'

Environment

VMware vCenter Server 8.x

Cause

• The Machine SSL certificate chain is invalid, or intermediate certificates are missing from the Machine_SSL_Cert or TRUSTED_ROOTS store. If any certificate in this hierarchy is missing from the chain (whether it is Certificate B, Certificate C, or the Root Certificate), the chain of trust breaks, and certificate verification fails.
• Successful certificate verification requires the entire chain of trust to be present. Below is an example of a complete hierarchy using multiple intermediate CA certificates:
  • Machine SSL Certificate (Signed by Intermediate Certificate B)
  • Intermediate Certificate B (Signed by Intermediate Certificate C)
  • Intermediate Certificate C (Signed by Root Certificate)
  • Root Certificate (Self-signed trust anchor)

Resolution

Resolving a Missing CA Certificate:

1. Identify the missing certificate by reviewing the pre-check error, which displays the subject of the missing CA certificate. The error appears similar to the following: Unable to find the root certificate with the subject 'X509Name object '/C=US/ST=#####/L=#####/O=######.com, Inc./CN=#######''
2. Import the missing certificate using any of the following three methods:
   • Using the vSphere Client: Add a Trusted Root Certificate to the Certificate Store Using the vSphere Client
   • Using the CLI Utility: Request Certificates and Import a Custom Root Certificate Using the CLI
   • Using the vCert Script: vCert
     • Run vCert script and navigate to Option 3, Manage certificates > Option 3. CA certificates in VMware Directory > Publish CA certificate(s) to VMware Directory.

To fix a MACHINE_SSL_CERT store that contains incorrect trusted roots or is missing the trusted root certificate in the chain:

1. Extract the current certificate from the MACHINE_SSL_CERT store: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/machine_ssl.crt
   
   
2. Extract the current private key: /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/machine_ssl.key
   
   
3. Build the complete certificate chain. (Refer to: How to correctly chain custom root/intermediate SSL certificates to import into vCenter).
   
   Open the extracted certificate file (/var/tmp/machine_ssl.crt) in a text editor and append the intermediate and root certificates. The file must contain the complete chain in the exact order below:
   
   • Machine SSL Certificate
   • Intermediate CA Certificate(s)
   • Root CA Certificate
4. Delete the existing Machine SSL entry: /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT -y
   
   
5. Add the entry back using the updated full chain: /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/machine_ssl.crt --key /var/tmp/machine_ssl.key
   
   
6. Force a refresh of VECS by running the following command: /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
   
   
7. Restart the vCenter Server services to apply the changes: service-control --stop --all && service-control --start --all
   
   
8. Re-run the upgrade after making these changes.

Additional Information

Sample steps to export the CA Certificate from local Windows Desktop:

Note: This is just an example to show how to export CA certificate from the Certificate Path based on issuer name ("Go Daddy Root Certificate Authority - G2" in this example).

1. Open the Machine SSL of Certificate of vCenter Server from local Windows Desktop.
   
   
2. Click on Certificate Path tab, select the CA certificate to export and Click on View Certificate.
   
   
   
   
3. A new window opens for the selected CA certificate from the Certificate Path.
   
   
   
   
4. Select the Details tab of the CA Certificate and click on Copy to File.
   
   
   
   
5. Click Next on the new window to export the Certificate.
   
   
   
   
6. Select Base-64 encoded X.509 (.CER).
   
   
   
   
7. Enter the Filename for the exported CA certificate and Click Next.
   
   
   
   
8. Click Finish to save the Certificate