Configuring CA signed certificates for ESXi hosts

Products

VMware vSphere ESXi

Issue/Introduction

This article discusses the configuration of Certificate Authority (CA) certificates for a ESXi 6.x/7.0 host. The instructions provided help eliminate errors or common causes for problems during certificate implementation, including configuration steps and details, and avoid misconfiguration in implementation of custom certificates in the virtual environment.

ESXi hosts that are upgraded from vSphere 5.x to vSphere 6.x continue to use their CA signed certificates if they were replaced in the previous versions. However, ESXi 5.x hosts that were running self-signed certificates and then upgraded to vSphere 6.x will have their certificates regenerated using VMware-signed. For more information, see Host Upgrades and Certificates in the vSphere Security guide. This article will help provide the following:

Steps to replace ESXi SSL custom certificate
Steps to add/install/change and generate an ESXi host certificate
Steps import and replace certificate for an ESXi custom certificate

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Resolution

Note: This article is part of several resolution paths. Before performing the steps in this article, see Replacing default certificates with CA signed SSL certificates in vSphere 6.x https://broadcomcms-software-agent.wolkenservicedesk.com/wolken/esd/knowledge-base-view/view-kb-article?articleNumber=343739&isLocationBackOnHome=true&hideTabs=true

Creating CA assigned certificates for an ESXi 6.x/7.0 host is a complex task. In many organizations, it is required to maintain proper security for regulatory requirements. Each server must be unique to the component as it ties to the fully qualified domain name of the server. Consequently, a single certificate cannot be applied it to all hosts. Wildcard certificates are currently NOT supported, but even if they were, it is much more secure to have a proper certificate for each host. There are several different workflows required for a successful implementation:

Generating the certificate request
Getting the certificate
Installation and configuration of the certificate on the ESXi host

These steps must be followed to ensure successful implementation of a custom certificate for an ESXi 6.x/7.0 host. Before attempting these steps ensure that:

This is vSphere 6.x/7.x/8.x environment.
The article Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.x/7.x (315271) has been reviewed.
There is an SSH client (such as Putty) installed
There is access to an SFTP/SCP client (such as WinSCP)

Generating a Certificate Request

To generate a certificate request for an ESXi 6.x/7.0 host:

Open a command prompt and navigate to the OpenSSL directory as previously configured using the Configuring OpenSSL for installation and configuration of CA signed certificates in the vSphere environment (344117) article. By default this is C:\OpenSSL-Win32\bin.

Run the command from the vcsa ssh shell:

1. /usr/lib/vmware-vmca/bin/certool --genkey --privkey=/root/esxiX.key --pubkey=/root/esxiX.pub

2. /usr/lib/vmware-vmca/bin/certool --gencsr --privkey=/root/esxiX.key --pubkey=/root/esxiX.pub --csrfile=/root/esxiX.csr --config=/dev/null --Country="US" --Name="esxiX.acme.com" --Organization="Acme" --OrgUnit="Virtual Infrastructure" --State="California" --Locality="Palo Alto" --IPAddress="10.0.0.1" --Hostname="esxiX.acme.com,esxiX"

This corrected the castore.pem file with only the intermediate and the root certificates, we then saved and rebooted and reconnected to vCenter.

Getting the Certificates

After the certificate request is created, the certificate must be given to the certificate authority for generation of the actual certificate. The authority presents a certificate back, as well as a copy of their root certificate, if necessary. For the certificate chain to be trusted, the root certificate must be installed on the server.

Follow the appropriate section for the steps for the certificate authority in question.

For Commercial CAs:

Take the certificate request ( rui.csr, as generated above) and send it to the authority in question.
The authority will send back the generated certificate.
Install the root certificate in vCenter Server before proceeding to the Installation of the certificate section of this document.

For Microsoft CAs:

Note: For Windows Server 2003 CA's, Enterprise edition is required. Other Windows Server 2003 editions do not have the correct templates for exporting a valid SSL certificate.

Create a custom template for certificate creation. For more information, see Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.x/7.x (315271)
Log in to the Microsoft CA certificate authority web interface. By default, it is http://servername/CertSrv/ .
Click Request a certificate.
Click advanced certificate request.
Click Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.
Open the certificate request using a text editor.
Copy the content from -----BEGIN CERTIFICATE REQUEST----- to -----END CERTIFICATE REQUEST----- into the Saved Request box.
Select the custom template created in step 1.
Click Submit to submit the request.
Click Base 64 encoded on the Certificate issued screen.
Click Download Certificate.
Save the certificate on the desktop of the server as rui.crt. When complete, proceed to Installing and configuring the certificate on the ESXi host to complete the configuration of the custom certificate.

Installing and configuring the certificate on the ESXi host

After the certificate is created, complete the installation and configuration of the certificate on the ESXi 6.x/7.0 host:

Note: Skip the first two steps if the ESXi is not part of vCenter Server (standalone host). Also, steps mentioned in "Process to update the vCenter Server database with the new certificate thumbprint" is not applicable for Standalone ESXi hosts.

Log in to vCenter Server.
Put the host into Maintenance Mode.

Note: If the ESXi 6.x/7.0 connects to VC 6.x/7.0 after replacing its cert with this KB, the host certs will be replaced by VMCA signed certs. VC 6.x/7.0 needs to switch to custom certificate mode first. For more information, see ESXi Certificate Mode Switch Workflows and Change the ESXi Certificate Mode.
Navigate to the console of the server to enable SSH on the ESXi host
Press F2 to log in to the Direct Console User Interface (DCUI)
Click Troubleshooting options > Enable SSH
Log in to the host and then navigate to /etc/vmware/ssl
Copy the files to a backup location, such as a VMFS volume
Log in to the host with WinSCP and navigate to the /etc/vmware/ssl directory
After backing up the existing rui.crt and rui.key, delete them from the directory
Copy the newly created rui.crt and rui.key to the directory using Text Mode or ASCII mode to avoid the issue of special characters ( ^M) appearing in the certificate file
Add the CAs in the signing chain of the new cert to the /etc/vmware/ssl/castore.pem directory
Type vi rui.crt to validate that there are no extra characters

Note: There should not be any erroneous ^M characters at the end of each line.
Switch back to the DCUI of the host and select Troubleshooting Options > Restart Management Agents
When prompted press F11 to restart the agents. Wait until they are restarted
Press ESC several times until you logout of the DCUI
Exit the host from Maintenance Mode

When complete, the host is made available and successfully rejoins the cluster.

Note: If the host is a part of a View cluster, you may need to perform these steps after updating the certificates to update the vCenter Server database with the new certificate thumbprint:

Process to update the vCenter Server database with the new certificate thumbprint:

Log in to vCenter Server
Place the host into the Maintenance Mode
Right-click the host and click Disconnect
Remove the disconnected host from the View cluster
Recompose the View desktop(s) again on the existing hosts in the cluster and ensure that they recompose successfully
Right-click on the disconnected host and select Connect
Add the host back to the cluster
Set DRS to Manual (optional)
Recompose the desktop on the host that was recently added back into the cluster
If step 9 is successful, set DRS back to Automatic, if required

The configuration of the custom certificate is now complete. Repeat these steps for each host which needs to have a custom certificate.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
If the error "Failed to import new SSL certificate" is displayed, please verify the MD5 check against the Private Key and the Certificate files, both should match if the Private Key belongs to the same certificate.

Login to ESXi host using SSH
Use below commands to run MD5 check

openssl x509 -in <path to the Certificate file> -noout -modulus | openssl md5

openssl rsa -in <path to the Private Key file> -noout -modulus | openssl md5

Example (where certificate not matching with private key):
# openssl x509 -in rui.crt -noout -modulus | openssl md5
(stdin)= 55d84795791549fe72fc498c69f0dd2d
# openssl rsa -in rui.key -noout -modulus | openssl md5
(stdin)= 6b84b1c62e91dbfc6b9f9efa5d34fb86

Reviewing hostd log (/var/run/log/hostd.log) will help to identify the cause if the MD5 values are matching and still getting the same error during Import operation.

Configuring CA signed certificates for ESXi 5.x hosts
Replacing default certificates with CA signed SSL certificates in vSphere 6.x
Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.0