"Error: 503 Service Unavailable" unable to access ESXi host client or add host to vCenter after host certificate update

Products

• VMware vSphere ESXi

Issue/Introduction

• Attempting to access ESXi UI after configuring CA signed certificates for ESXi receiving:
  
  Error: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x0000001209e60f60] _serverNamespace = / action = Allow _port = 8309)" 

  
  
• If the host is not currently managed by a vCenter, attempts to add it to the vCenter fail with this message in the vpxd.log:
  
  Failed to get pooled connection; <cs p:00007f4348243440, TCP:esxi.host.fqdn:443>, (null), duration: 2msec, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

• The hostd log (/var/log/hostd.log) contains a message similar to:
  
  [Originator@6876 sub=Solo] Initializing keystore failed: N7Vmacore6Crypto15CryptoExceptionE(Crypto Exception: error:0906D06C:PEM routines:PEM_read_bio:no start line)

Please check the validity of certificate files /etc/vmware/ssl/rui.key and /etc/vmware/ssl/rui.crt
  
  OR 
  
  [Originator@6876 sub=Solo] Failed to create SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:05800074:x509 certificate routines::key values mismatch)
  
  
• The vpxa log (/var/log/vpxa.log) contains a message similar to:
  
  [Originator@6876 sub=Default] Failed to initialize the SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch) --> Panic: Failed to initialize the SSL context.
  
  
• The hostd and vpxa services will not start, or will start and then stop.
• rui.crt and rui.key files are missing from /etc/vmware/ssl, or they are present but contain errors.

Environment

VMware vSphere ESXi

Cause

When an invalid SSL certificate key pair /certificate is uploaded through the vSphere client or added manually through the CLI on an ESXi host, it's refused but applied nevertheless, crashing any and all of the management daemons.

Resolution

To resolve the issue,

• If rui.crt and rui.key files are present, create a backup directory for them and move them to the backup directory.
  
  cd /etc/vmware/ssl/
mkdir backup
mv rui.crt ./backup/
mv rui.key ./backup/
  
  
• Regenerate the self-signed certificate by executing the following command:
  
  /sbin/generate-certificates

• Restart the management agents (you may also need to restart the ESXi host):
  
  /etc/init.d/hostd restart
/etc/init.d/vpxa restart
  
  
• Verify the check against the Private Key and the Certificate files before configuring CA signed certificate for ESXi, both should match if the Private Key belongs to the same certificate:
  
  openssl x509 -noout -modulus -in rui.crt | openssl sha256
openssl rsa -noout -modulus -in rui.key| openssl sha256