Configuring CA signed certificates for ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

This article outlines the configuration of Custom Certificate Authority (CA) signed certificates for an ESXi host. The provided instructions aim to mitigate common implementation issues and ensure proper configuration details are followed, thereby preventing misconfiguration of custom certificates within the environment.

Environment

• VMware vSphere ESXi 5.x
• VMware vSphere ESXi 6.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Resolution

Maintaining robust security is a critical requirement for organizations aiming to meet stringent regulatory standards. Given that each server is uniquely identified by its Fully Qualified Domain Name (FQDN), deploying a distinct digital certificate for every component is a necessary operational practice.

Applying a single certificate across all hosts is not an acceptable security practice. While wildcard certificates are not currently supported, utilizing a dedicated, unique certificate for each host is considered the most secure and highly recommended approach. This method reinforces security posture and aids in clear, distinct host identification.

Successful implementation of this security measure involves several distinct workflows:

• Creating the certificate request
• Obtaining the certificate
• Installing and configuring the certificate on the ESXi host

Step 1: Generating a certificate request

To generate a certificate request for an ESXi host:

1. Login to ESXi via SSH. Refer to Enabling ESXi Shell access using the vSphere Client
2. Create a Template file with the required parameters under /tmp using vi editor
   
   vi /tmp/openssl.cfg

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:<ESXi_Shortname>, IP:<ESXi_IP>, DNS:<ESXi_FQDN>

[ req_distinguished_name ]
countryName = <Country_code>
stateOrProvinceName = <State>
localityName = <Locality>
0.organizationName = <Organization_Name>
organizationalUnitName = <OU>
commonName = <ESXi_FQDN>

Note: Replace the code in bold with the details of the system requiring the SSL certificates: 

     2.Execute the command:

openssl req -new -nodes -out /tmp/rui.csr -keyout /tmp/rui.key -config /tmp/openssl.cfg

This creates the certificate request rui.csr in /tmp

Step 2: Getting the signed certificate

Once the Certificate Signing Request (CSR) is generated, it must be submitted to the Certificate Authority (CA). The CA processes the request and subsequently provides the signed certificate. They may also include a copy of their root certificate for installation, if needed for trust establishment.

 

Follow the appropriate section below for the steps for the certificate authority in question.

For Commercial CAs:

1. Submission of CSR: Submit the generated Certificate Signing Request (CSR) file (e.g., rui.csr) to the designated Certificate Authority (CA).

2. Certificate Retrieval: Upon processing, retrieve the signed certificate issued by the CA.

3. Root Certificate Installation: Before proceeding to the final installation of the signed certificate, the CA's root certificate must be installed on the vCenter server.

For Microsoft CAs:

Log in to the Microsoft CA certificate authority web interface. By default, it is http://<servername>/CertSrv/

1. Click Request a certificate.
2. Click advanced certificate request.
3. Click Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.
4. Open the certificate request in a plain text editor.
5. Copy from -----BEGIN CERTIFICATE REQUEST----- to -----END CERTIFICATE REQUEST----- into the Saved Request box.
6. Click Web Server when selecting the Certificate Template.
7. Click Submit to submit the request.
8. Click Base 64 encoded on the Certificate issued screen.
9. Click Download Certificate.
10. Save the certificate on the desktop of the server as rui.crt.

Step 3: Installing and configuring the certificate on the ESXi host

After the certificate is created, complete the installation and configuration of the certificate on the ESXi host:

1. Log in to vCenter Server
2. Put the host into Maintenance Mode.
3. Log in to the SSH of the host and then navigate to /etc/vmware/ssl .
4. Move the files to a backup location
   
   mv /etc/vmware/ssl/rui.* /var/core/
   
   
5. Copy the existing rui.key to the same directory
   
   mv /tmp/rui.key /etc/vmware/ssl/rui.key
   
   
6. Log in to the host with WinSCP and navigate to the /etc/vmware/ssl directory
7. Replace the newly created certificate to the /etc/vmware/ssl/ directory and rename as rui.crt
   Note: There should not be any erroneous ^M characters at the end of each line.
8. Navigate to the DCUI of the host. Refer Accessing Direct Console User Interface
9. Select Troubleshooting Options > Restart Management Agents.
10. When prompted press F11 to restart the agents. Wait until they are restarted.
11. Press ESC to logout of the DCUI.
12. The host would be in disconnected state on the vCenter 

Step 4: Change certificate mode of vpxd

Techdoc: Change the ESXi Certificate Mode

 

To implement custom certificates from an external Root Certificate Authority (CA), the advanced vCenter Server setting, vpxd.certmgmt.mode, must be modified. Following this change, hosts will no longer be automatically provisioned with VMware Certificate Authority (VMCA) certificates upon certificate refresh. All subsequent certificate lifecycle management for the environment becomes the administrative responsibility. The advanced settings permit configuration for either Custom CA mode or Thumbprint mode; Thumbprint mode should be utilized exclusively as a fallback option.

1. In the vSphere Client, select the vCenter Server system that manages the hosts.
2. Click Configure, and under Settings, click Advanced Settings.
3. Click Edit Settings.
4. Click the Filter icon in the Name column, and in the Filter box, enter vpxd.certmgmt to display only certificate management parameters.
5. Change the value of vpxd.certmgmt.mode to custom if administrative staff intends to manage their own certificates, or to thumbprint if temporary thumbprint mode is required, and then click Save.

Step 5: Reconnect the ESXi host

Right Click the ESXi Server and navigate to Connection -> Connect

Note: If the HA election fails, timeout, or waiting to join, then disconnect and reconnect all hosts from the cluster. A new election will occur using the updated SSL thumbprints.

The configuration of the custom certificate is now complete. Repeat these steps for each host which needs to have a custom certificate.

Additional Information

• After upgrading to vSphere 5, you see the HA error: vSphere HA Cannot be configured on this host because its SSL thumbprint has not been verified
• Configuring OpenSSL for installation and configuration of CA signed certificates in the vSphere environment
• Configuring CA signed certificates for ESXi 6.0 hosts