Upgrading vCenter Server or ESXi 8.0 fails during precheck due to a weak certificate signature algorithm

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Upgrading to vCenter Server or ESXi 8.0 requires an additional precheck. The precheck ensures that vCenter Server and ESXi are not using certificates with weak signature algorithms, such as sha1WithRSAEncryption. Certificates with weak signature algorithms are no longer supported in vSphere 8.0 in most contexts.

vCenter Server upgrades may fail during precheck with an error stating, "Support for certificates with weak signature algorithms has been removed from vCenter Server 8.0." 



ESXi upgrades may fail during precheck with an error string stating, "Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0." 

 

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Resolution

Replacing Certificates

The vCenter Server Machine Certificate

If the upgrade precheck failure message indicates that a problematic certificate is present in the VECS store "MACHINE_SSL_CERT", then the vCenter Server machine certificate must be replaced before upgrade can proceed. The vCenter Server machine certificate is the primary TLS certificate used by the vCenter Server HTTPS endpoint, which defaults to port 443. This is the certificate that is seen when connecting to the vCenter Server web UI.

Steps to replace the certificate:

1. Replace the vCenter Server machine certificate, and the associated root and intermediate certificates with a certificate that contains a SHA-2 digital signature

• Certificate replacement via vSphere UI
• Certificate replacement via certificate-manager CLI
• Certificate management REST API

2. Push the new CA certificate(s) to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.

• Refresh ESXi CA certificates via vSphere UI

3. Remove the old and now unused root certificate from vCenter Server using the `dir-cli trustedcert unpublish` command.

• Remove certificates using dir-cli
• Certificate management REST API

A vCenter Server Trusted Root Certificate

If the upgrade precheck failure message indicates that a problematic certificate is present in the VECS store "TRUSTED_ROOTS", then vCenter Server has configured trusted root or intermediate certificate that must be removed or replaced before upgrade can proceed. There may be a dependency on the problematic trusted root certificates and it's important to update the dependent services before removing or replacing the certificate. Also note that the certificates that are present in the VECS store "TRUSTED_ROOTS" are pushed to all connected ESXi hosts.

Steps to replace the certificate:

1. Add a new certificate to the "TRUSTED_ROOTS" store that will replace the now unsupported certificate.

• Add certificate via the vSphere UI
• Add certificates using `dir-cli trustedcert publish`
• Certificate management REST API

2. Push the new CA certificate(s) to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.

• Refresh ESXi CA certificates via vSphere UI

3. Ensure that any certificate signed by the unsupported certificate is removed from the vSphere environment. If a leaf certificate is found to be signed by the unsupported certificate, it must also be replaced. Consider the following cases.

• The leaf certificate may be in use on an ESXi host.
• The leaf certificate may be used by a VMware solution, such as vRA, vROps, SRM, NSX, etc.
• The leaf certificate may be used by a partner solution, such as vVols or a backup solution.

Note: For vVols please re-register the VASA Provider in vCenter

4. Once the unsupported certificate is no longer in use, it may be removed from the TRUSTED_ROOTS store.

• Remove certificates using dir-cli
• Certificate management REST API

5. Push the CA certificate changes to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.

• Refresh ESXi CA certificates via vSphere UI

VMCA is Acting as a Subordinate CA

In a default vCenter Server deployment, VMCA is the root certificate authority (CA). However, VMCA can be configured a subordinate CA where an outside authority is the root and VMCA uses an intermediate certificate. If VMCA is configured as a subordinate CA and the root certificate uses a weak digital signature, then the root certificate will need to be replaced and all certificates reissued.

1. Reset the VMCA certificate using one of the following methods:

• Reset the VMCA certificate using certificate-manager CLI

- A new intermediate certificate may be configured with VMCA as a subordinate CA.
- A default VMCA root certificate may be configured.
- All vCenter Server certificates will be regenerated.

• Reset the VMCA certificate using the REST API

- A default VMCA root certificate may be configured.
- All vCenter Server certificates will be regenerated.
- Note that the REST API cannot be used to configure VMCA as a subordinate CA.

2. Reissue all ESXi TLS certificates with the new VMCA certificate if they were signed with the previous VMCA certificate.

• Issue a VMCA signed TLS certificate for ESXi using the vSphere UI

3. Remove the old and now unused CA root certificate and intermediate certificates from the TRUSTED_ROOTS store of VECS.

• Remove certificates using dir-cli

4. Push the CA certificate changes to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.

• Refresh ESXi CA certificates via vSphere UI

vCenter Server BACKUP_STORE Certificate

If the upgrade precheck failure message indicates that a problematic certificate is present in the VECS store "BACKUP_STORE", then the certificate can be safely removed using the `vecs-cli` command.

• Remove the certificate using `vecs-cli`

The ESXi Server TLS Certificate (rui.crt)

The ESXi TLS certificate is managed by vCenter Server by default, however administrators may choose to manually assign a certificate. If the current TLS certificate contains a weak digital certificate, then a new certificate must be issued. Note that this certificate is stored in a file name "rui.crt" which may be displayed in the upgrade precheck error messages.

• Issue a VMCA signed TLS certificate for ESXi using the vSphere UI
• Issue a custom TLS certificate for ESXi

The ESXi Server Certificate Store (castore.pem)

vCenter Server pushes its own Trusted Root certificates, "TRUSTED_ROOTS", to the ESXi Certificate Store. However, each ESXi host may have additional certificates added manually as well. If the ESXi Certificate Store contains a certificate with a weak digital signature then the certificate can be removed using esxcli.

The currently configured certificates can be listed with the following command.
$ esxcli system security certificatestore list

Certificates can be used with the following command.
$ esxcli system security certificatestore remove

Note that esxcli command can also be accessed using PowerCLI

Standalone Precheck Script

The `vsphere8_upgrade_certificate_checks.py` Python script verifies that vCenter Server and the connected ESXi hosts are not using certificates with a weak digital signature algorithm. This is a standalone version of the same prechecks that are preformed during vCenter Server and ESXi upgrades. It can be run manually before a planned upgrade maintenance window.

The script first checks if vCenter Server has any unsupported certificates in the VECS stores. It then iterates through all of the ESXi hosts in the inventory to perform similar checks.

To run the script perform the following steps:

1. Download the `vsphere8_upgrade_certificate_checks.py` script attached to this KB.
2. Transfer the script file to a temporary folder (e.g. `/tmp`) on vCenter Server (e.g. using `scp` or `WinSCP`).
3. Login to the vCenter Server appliance using root credentials with an SSH client, or create a new RDP session if the vCenter Server is Windows based.
4. Execute the script using the following command:

• On Linux using shell: `python /tmp/vsphere8_upgrade_certificate_checks.py`
• On Windows using cmd.exe: `"%VMWARE_PYTHON_BIN%" C:\Users\Administrator\Desktop\vsphere8_upgrade_certificate_checks.py`

If any certificates with a weak signature algorithm are found, the details are printed to the console window. These issues should be resolved before proceeding with upgrade. An example output with failures is shown below.

vVols (Virtual Volumes)

vVol VASA Provider uses Self Signed Certificate that are issued by 3rd party VASA providers which in certain cases are stored directly in VC trusted store. Follow the below mentioned steps:

1. Verify if the certificate algorithm is of sha1WithRSAEncryption from “Certificate Info” section of VASA provider. If So, follow below steps to update Certificate to SHA256.
2. Confirm the self-signed certificate using the following command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

Alias : https://<VP_IP>:<PORT_NO>/version.xml
Signature Algorithm: sha1WithRSAEncryption
SAN: IP:VP_IP, DNS:VP_DNS
CA:False

3. If CA field is 'False', then it’s a self-signed certificate and if the user is using vVols, then a new self signed certificate needs to be generated and installed on VASA provider which has at least SHA256 Signature.

Note – While installing the certificates the VASA provider may be unavailabile on ESXi host.

4. Unregister/re-register the VASA provider in all VCs that are connected to this VASA provider.
5. Run the root sync so all the connected hosts get the new trusted root.
6. Proceed with the upgrade.