ESXi host is automatically disconnected from vCenter with such error "Failed to verify heartbeat signature" in vpxd log in vCenter

Products

• VMware vSphere ESX 7.x

Issue/Introduction

ESXi host is in vCenter inventory but automatically disconnected from vCenter. After reconnecting, the following situations may occur,

1. Reconnecting is failed and the host is still in 'disconnected' state

2. Reconnecting is successfully, but after a period of time, the same issue occurs again, leading to another disconnection.

In vCenter we see "Failed to verify heartbeat signature" error for that host in vpxd log,

2025-01-07T15:59:21.892+08:00 error vpxd[06409] [Originator@6876 sub=vpxCrypt opID=HeartbeatModuleStart-53d3d42] [bool VpxPublicKey::Verify(const EVP_MD*, const unsigned char*, size_t, const unsigned char*, size_t) const] openssl error: error in EVP_VerifyFinal()
2025-01-07T15:59:21.892+08:00 error vpxd[06409] [Originator@6876 sub=vpxCrypt opID=HeartbeatModuleStart-53d3d42] [bool VpxPublicKey::Verify(const EVP_MD*, const unsigned char*, size_t, const unsigned char*, size_t) const] ERR error:02000077:rsa routines::wrong signature length
2025-01-07T15:59:21.892+08:00 error vpxd[06409] [Originator@6876 sub=vpxCrypt opID=HeartbeatModuleStart-53d3d42] [bool VpxPublicKey::Verify(const EVP_MD*, const unsigned char*, size_t, const unsigned char*, size_t) const] ERR error:1C880004:Provider routines::RSA lib
2025-01-07T15:59:21.892+08:00 warning vpxd[06409] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-53d3d42] Failed to verify heartbeat signature; [vim.HostSystem:host-13934,192.168.0.35], cert: 9E:58:15:66:5D:24:07:D4:D2:E4:2D:48:99:C4:33:99:16:3F:18:C8, signature: aehfu7+7sqqwe4JbUikkP5V3RoC0rcrkgedn3XLvt6YnZw6l6qksUdjYCgwoTkQBkBQn24/4J6Fq365s7/UETPMV5mu/XJk048Xt6Knp46Yr/Jy0rzNzCjDCrBBRQniwQSb84GMkgfmh+pP7RNV8MzEhNNcrmW7daswUMJhrcLQ/4MMy+bauedoZD2Ek5urXGeZuZ7afgc4oljzqn4ewR8BVbMxsv4dbrtM755XWaC+KX1Ugd088ssycsjNX4EsgW9HGMFqsqXKpFxUlWEC4mXVar71EfQbCsCcOXA1nzPjTe0knepzw85h2J7DQLcQEBfjTIqCNolOFnzQcI9clAQ==, msg: {srv: 49375, gen: 7738, ct: 8237, bld: 23794027, cnx: 52d4b833-73d5-ab2a-e56b-2837219252b9, ip: 192.168.0.35}

Note. The timestamps and IP address of the host are for example purpose.

Environment

The vCenter is in a different subnet from the management network of ESXi host, with a firewall and cloud/network management services in between.

Cause

The certificate of ESXi host appears differently on the ESXi host itself and vCenter.

 

Resolution

On the host we can check the certificate fingerprint by either of the two commands (replace the ESXi_IP_ADDRESS with real IP address), we should see the same output,

[root@esxi-host:~] cat /etc/vmware/ssl/rui.crt |openssl x509 -fingerprint -sha1 -noout
sha1 Fingerprint=EE:84:5F:71:8B:99:A0:64:79:01:DB:D4:FC:8C:B0:6E:FC:77:BC:3B

[root@esxi-host:~] openssl s_client -connect ESXi_IP_ADDRESS:443 </dev/null 2>/dev/null | openssl x509 -fingerprint -sha1 -noout
sha1 Fingerprint=EE:84:5F:71:8B:99:A0:64:79:01:DB:D4:FC:8C:B0:6E:FC:77:BC:3B

However from vCenter we see a totally different fingerprint of the ESXi certificate,

root@sa-vcsa-01 [ ~ ]# openssl s_client -connect ESXi_IP_ADDRESS:443 </dev/null 2>/dev/null | openssl x509 -fingerprint -sha1 -noout
sha1 Fingerprint=0B:08:8C:A5:D7:C4:B4:97:C3:BB:B0:7A:75:F9:17:C7:01:D6:CE:97

We can execute the following command on ESXi host and vCenter respectively to check the full details of the certificate on ESXi host. This will allow us to identify where the differences lie.

openssl s_client -connect ESXi_IP_ADDRESS:443

Through this check, we can confirm that the certificate of ESXi host appears differently on the ESXi host itself and vCenter. This is most likely due to modification by the firewall or cloud/network management software in between.