HA configuration stuck in Election state due to host SSL certificate thumbprint mismatch

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• vSphere HA configuration stays in the "Election" state and does not proceed for a group of hosts.
• Following the renewal of the host SSL certificate, vSphere HA agent shows Unreachable and the election process is failing.
• FDM logs on ESXi shows the errors as below:
  
  /var/run/log/fdm.log

YYYY-MM-DDTHH:MM warning fdm[#####] [Originator@6876 sub=Cluster opID=SWI-#####] Failed to verify host  (###.###.###.###) - closing connection
YYYY-MM-DDTHH:MM verbose fdm[####] [Originator@6876 sub=Message opID=SWI-#####] Accept completion callback error N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError--> )

YYYY-MM-DDTHH:MM:SS.###Z Db (167) Fdm[#####]: [Originator@6876 sub=Cluster opID=WorkQueue-######] IP ###.###.###.### marked bad for reason Unreachable IP
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: [Originator@6876 sub=IO.Connection opID=WorkQueue-2bb6be0] Failed to SSL handshake; SSL(<io_obj p:0x0000009e9083c7d0, h:8<TCP '###.###.###.###:8182'>>), e:167772294 (certificate verify failed (SSL routines) ),duration: 1msec
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: [Originator@6876 sub=Message opID=WorkQueue-2bb6be0] Error N7Vmacore3Ss118SSLVerifyExceptionE (SSL Exception: Verification
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: -- > PeerThumbprint: 3F:##:##:##:##:##:##:##:##:##:##:##:DA:##:##:##:##:##:##:##:##:##:##:BB
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: -- > ExpectedThumbprint: FF:##:##:##:##:##:##:##:##:##:##:2B:##:##:##:##:##:##:##:##:##:##:##:AC
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: -- > ExpectedPeerName: ########
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: -- > The remote host certificate has these problems:
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fom[#####]: -- >
YYYY-MM-DDTHH:MM:SS.###Z Er (163) Fdm[#####]: -- > * Host name does not match the subject name (s) in certificate.)

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

• This occurs due to mismatch of SSL thumbprints between the vCenter Server database (VCDB) and the actual host SSL certificate. Such a mismatch could occur if the SSL certificate replacement uses methods other than those described in Techdocs Replacing the Default ESXi Certificate with a Custom Certificate.
• Follow these steps to verify the SSL thumbprint stored on vCenter Server and the actual SSL thumbprint of the host:
  • Check SSL thumbprint on vCenter Server
    • Login to vCenter Server via SSH.
    • Execute this command to list the host SSL thumbprint stored on VCDB
      
      psql -U postgres -d VCDB -c "select id,dns_name,ip_address,host_ssl_thumbprint,expected_ssl_thumbprint from vpx_host;"
      
      
  • Check SSL thumbprint on the ESXi host:
    • Login to ESXi host via SSH
    • Execute this command to list the host SSL thumbprint of ESXi host certificate
      
      openssl x509 -in /etc/vmware/ssl/rui.crt -text -fingerprint |grep -i fingerprint

Resolution

Disconnect and reconnect the impacted hosts to update the host's current SSL thumbprint in vCenter Server. Refer Disconnecting and Reconnecting an ESXi Host for more details.