ESXi host client Shows “no healthy upstream” error after replacing custom certificate

Products

• VMware vSphere ESXi

Issue/Introduction

• The ESXi host is shown as disconnected in the vCenter UI, reconnection attempts fail with the error message listed below, and the ESXi host client reports a “no healthy upstream” error.
  
  "Cannot contact the specified host (<hostname>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."


• The ESXi management agents (hostd and vpxa) crash during initialization, and the following entries are observed in the /var/run/log/hostd.log file on the ESXi host.
  
  YYYY-MM-DDTHH:MM:SS.###Z In(###) Hostd[######]: [Originator@#### sub=Solo] Setting OpenSSL verify location CAFile=/etc/vmware/ssl/castore.pem
YYYY-MM-DDTHH:MM:SS.###Z Er(###) Hostd[######]: [Originator@#### sub=Solo] Failed to create SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:########:x509 certificate routines::key values mismatch)

Environment

VMware vSphere ESXi 8.x

Cause

This issue occurs due to a cryptographic mismatch between the certificate file (rui.crt) and the private key (rui.key), where comparison of the SHA-256 thumbprints confirms that the private key does not correspond to the installed certificate.

Resolution

• Obtain the correct rui.crt and rui.key files before proceeding.
  
  
• Create a backup of the existing rui.crt and rui.key files.
  
  • mv /etc/vmware/ssl/rui.crt /etc/vmware/ssl/rui.crt.bkp
  • mv /etc/vmware/ssl/rui.key /etc/vmware/ssl/rui.crt.bkp
    
    
• Proceed with updating the custom certificate on the ESXi host by referring the below KB article.
  •  Adding Custom Certificate on ESXi hosts through CLI 
    
    
• Run the following openssl commands to verify that the SHA-256 thumbprints of rui.crt and rui.key match
  
  • openssl x509 -in rui.crt -pubkey -noout -outform pem | sha256sum
  • openssl pkey -in rui.key -pubout -outform pem | sha256sum
    
    
• Start the ESXi management services (hostd and vpxa).
  
  • /etc/init.d/hostd start
  • /etc/init.d/vpxa start