ESXi Host Disconnected from vCenter and hostd Fails to Start After SSL Certificate Replacement
Article ID: 367381
Updated On:
Products
Issue/Introduction
After replacing SSL certificates on an ESXi host following the procedure in VMware KB 56441 "Adding Custom Certificate on ESXi hosts through CLI", the host may experience the following issues upon reboot:
- The hostd service fails to start automatically
- The host does not connect to vCenter Server
- Manually starting the hostd service does not resolve the connectivity issue
These symptoms can lead to production downtime and impact mission-critical applications running on the affected host.
Environment
- ESXi host SSL certificates replaced using the procedure in KB 56441
Cause
The issue is likely caused by an incomplete execution of the certificate replacement procedure outlined in KB 56441. If the steps are not followed precisely, particularly the renaming of the old rui cert and key and updating the castore.pem file, the SSL certificate configuration can become corrupted.
A malformed or out-of-sync castore.pem file can lead to hostd authentication failures and prevent the host from connecting to vCenter Server.
Resolution
If you encounter a corrupted castore.pem file with a malformed hash at the beginning after an incomplete SSL certificate replacement, follow these steps to resolve the issue and restore connectivity between the affected ESXi host and vCenter Server:
1. Connect to the ESXi host via SSH or direct console.
2. Navigate to the SSL certificate directory: cd /etc/vmware/ssl/
3. Backup the existing castore.pem file: cp castore.pem castore.pem.backup
4. Open the castore.pem file for editing: vi castore.pem
5. Compare the contents of castore.pem with the root certificate (e.g., root.cer) used during the certificate replacement process.
6. Identify and remove any malformed hashes or extra content at the beginning of castore.pem that does not match the root certificate.
7. Save the changes to castore.pem and exit the text editor.
8. Verify the contents of the updated castore.pem file: cat castore.pem
9. Restart the hostd and vpxa services:
a. /etc/init.d/hostd restart
b. /etc/init.d/vpxa restart
10. Attempt to reconnect the host to vCenter Server.
If the issue persists or you are unsure about editing the castore.pem file, you can alternatively regenerate the certificate bundle by following these steps:
1. Connect to the ESXi host via SSH or direct console.
2. Navigate to the SSL certificate directory:
cd /etc/vmware/ssl/
3. Backup the existing castore.pem file:
cp castore.pem castore.pem.backup
4. Rename the current castore.pem file:
cp castore.pem castore.pem.old
5. Generate a new set of certificates, including castore.pem:
/sbin/generate-certificates
6. Verify the contents of the newly created castore.pem:
cat castore.pem
7. Restart the hostd and vpxa services:
a. /etc/init.d/hostd restart
b. /etc/init.d/vpxa restart
8. Attempt to reconnect the host to vCenter Server.
If custom SSL certificates are still desired, carefully follow all steps in KB 56441, paying special attention to replacing the rui cert and key to maintain the integrity of castore.pem and prevent authentication issues.
Additional Information
The example below shows a certificate from the castore.pem file with an invalid hash which demonstrates some of the types of problems you might see.
Notice the top-most hash contains:
- Non-hexadecimal characters like spaces, newlines, and other symbols
- Inconsistent capitalization of hexadecimal letters
- An incorrect hash length. It should be 64 characters for SHA-256.
Any of these issues would cause the certificate to be rejected as invalid by the ESXi system.
NOTE: these are not real certificate hashes - they're dummy hashes simply to illustrate what the file might look similar to
For more information about managing SSL certificates on ESXi hosts, refer to the following VMware documentation:
- vSphere and VCF Security Guide
- Configuring CA signed certificates for ESXi
Feedback
