ESXi Host Disconnected from vCenter and hostd Fails to Start After SSL Certificate Replacement

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

After replacing SSL certificates on an ESXi host following the procedure in Broadcom KB 317244 "Adding Custom Certificate on ESXi hosts through CLI", the host may experience the following issues upon reboot:

- The hostd service fails to start automatically
- The host does not connect to vCenter Server
- Manually starting the hostd service does not resolve the connectivity issue

These symptoms can lead to production downtime and impact mission-critical applications running on the affected host.

Environment

VMware ESXi Host

Cause

The issue is likely caused by an incomplete execution of the certificate replacement procedure outlined in Broadcom KB 317244. If the steps are not followed precisely, particularly the renaming of the old rui cert and key and updating the castore.pem file, the SSL certificate configuration can become corrupted.

A malformed or out-of-sync castore.pem file can lead to hostd authentication failures and prevent the host from connecting to vCenter Server.

Resolution

If you encounter a corrupted castore.pem file with a malformed hash at the beginning after an incomplete SSL certificate replacement, follow these steps to resolve the issue and restore connectivity between the affected ESXi host and vCenter Server:

1. Connect to ESXi Host:

   • Connect to the ESXi host via SSH or direct console.

2. Navigate to SSL Directory:

   • Navigate to the SSL certificate directory:

     cd /etc/vmware/ssl/
     

3. Backup castore.pem:

   • Create a backup of the existing castore.pem file:

     cp castore.pem castore.pem.backup
     

4. Open castore.pem for Editing:

   • Open the castore.pem file in a text editor (e.g., vi):

     vi castore.pem 
     

5. Compare with Root Certificate:

   • Compare the contents of castore.pem with the root certificate (e.g., root.cer) used during the certificate replacement process.

6. Remove Malformed Content:

   • Identify and remove any malformed hashes or extra content at the beginning of castore.pem that does not match the root certificate.

7. Save and Exit:

   • Save the changes to castore.pem and exit the text editor.

8. Verify castore.pem Contents:

   • Verify the contents of the updated castore.pem file:

     cat castore.pem

9. Restart Services:

   • Restart the hostd and vpxa services:

     /etc/init.d/hostd restart
     /etc/init.d/vpxa restart
     

10. Reconnect to vCenter Server:

    • Attempt to reconnect the host to vCenter Server.

 

 

If the issue persists, or you are unsure about editing the castore.pem file, you can alternatively regenerate the certificate bundle by following these steps:

1. Connect to ESXi Host:

   • Connect to the ESXi host via SSH or direct console.

2. Navigate to SSL Directory:

   • Navigate to the SSL certificate directory:

     cd /etc/vmware/ssl/

3. Backup castore.pem:

   • Create a backup of the existing castore.pem file:

     cp castore.pem castore.pem.backup
     

4. Rename castore.pem:

   • Rename the current castore.pem file:

     cp castore.pem castore.pem.old
     

5. Generate New Certificates:

   • Generate a new set of certificates, including castore.pem:

     /sbin/generate-certificates 
     

6. Verify castore.pem Contents:

   • Verify the contents of the newly created castore.pem:

     cat castore.pem
     

7. Restart Services:

   • Restart the hostd and vpxa services:

     /etc/init.d/hostd restart
     /etc/init.d/vpxa restart
     

8. Reconnect to vCenter Server:

   • Attempt to reconnect the host to vCenter Server.

 

If custom SSL certificates are still desired, carefully follow all steps in KB 317244, paying special attention to replacing the rui cert and key to maintain the integrity of castore.pem and prevent authentication issues.

Additional Information

The example below shows a certificate from the castore.pem file with an invalid hash, which demonstrates some types of problems you might see.

Notice the top-most hash contains:

• Non-hexadecimal characters like spaces, newlines, and other symbols
• Inconsistent capitalization of hexadecimal letters
• An incorrect hash length. It should be 64 characters for SHA-256.

Any of these issues would cause the certificate to be rejected as invalid by the ESXi system.

NOTE: these are not real certificate hashes — they're dummy hashes simply to illustrate what the file might look similar to

- The incomplete certificate replacement procedure can cause the /etc/vmware/ssl/castore.pem file to become corrupted, leading to hostd authentication failures and disconnection from vCenter Server.

- Regenerating the castore.pem file using the ESXi generate-certificates tool creates a fresh, valid certificate bundle and resolves the issue.
- If the customer still wishes to use custom certificates, emphasize the importance of following all steps in KB 317244 carefully to maintain the integrity of castore.pem.

For more information about managing SSL certificates on ESXi hosts, refer to the following VMware documentation:

- vSphere and VCF Security Guide
- Configuring CA signed certificates for ESXi