ESXis intermittently disconnected from vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• ESXi are getting disconnected intermittently from vCenter Server
• 3rd party backups are failing because of ESXis connectivity or SSL thumbprint errors
• In vCenter /var/log/vmware/vpxd/vpxd.log file, you can see lines similar to: 

<timestamp> warning vpxd[06067] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007f8244185e40, h:38, <TCP '<VCENTER_IP> : 39572'>, <TCP '208.91.112.55 : 443'>>), e: 167772294(certificate verify failed (SSL routines)), duration: 37msec
<timestamp> warning vpxd[06067] [Originator@6876 sub=HttpConnectionPool-000001] Failed to get pooled connection; <cs p:00007f820092dd70, TCP:<esxi_fqdn>:443>, SSL(<io_obj p:0x00007f8244185e40, h:38, <TCP '<VCENTER_IP> : 39572'>, <TCP '208.91.112.55 : 443'>>), duration: 50msec, N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: <thumbprint>
--> ExpectedThumbprint:
--> ExpectedPeerName: <esxi_fqdn>
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.
-->
--> * self-signed certificate)

• In vCenter /var/log/vmware/dnsmasq.log file, you can see lines similar to: 

<timestamp> dnsmasq[1485]: query[A] <esxi_fqdn> 127.0.0.1
<timestamp> dnsmasq[1485]: forwarded <esxi_fqdn> to <dns_server_ip>
<timestamp> dnsmasq[1485]: reply <esxi_fqdn> is 208.91.112.55

 

• IP 208.91.112.55 is not your esxi IP but Fortinet Firewall  default Redirect Portal IP

Environment

vCenter Server 8.x

Cause

• This is not a VMware issue

https://docs.fortinet.com/document/fortigate/6.2.16/cookbook/567703/fortiguard-category-based-dns-domain-filtering

If you select Block, there are two options:
Redirect Portal IP. If the DNS query domain will be blocked, FortiGate will use portal IP to replace the resolved IP in DNS response packet. You can use the default portal IP 208.91.112.55 or click Specify to enter another portal IP.
Block. Blocked DNS query has no response return and the DNS query client will time out.

 

Resolution

• Contact your firewall support for resolution
• As a workaround you can use add your esxis hosts in vCenter Server /etc/hosts file

Adding/Deleting/Editing a host entry on vCenter server or ESXi host using vi editor