Expired ESXi host certificates can impact vSAN functionality

Products

VMware vSAN

Issue/Introduction

Symptoms:

Expired ESXi host certificate(s) in a vSAN cluster can have negative impacts to vSAN functionality such as:

Incomplete unicast agent list on hosts resulting in unhealthy/inaccessible objects
esxcli vsan commands failing
The primary node not receiving performance data from other hosts in the cluster
vCenter/ESXi communication

In the /var/run/log/hostd.log file in the ESXi host, you see entries similar to:
<YYYY-MM-DD>T<TIME>Z error hostd[B182B70] [Originator@6876 sub=Default opID=378619de-c4-3001 user=vpxuser:com.vmware.vsan.health] AdapterServer caught exception: SSL Exception: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
<YYYY-MM-DD>T<TIME>Z info hostd[31240B70] [Originator@6876 sub=VsanSimsStubImpl opID=xxxxxxx-edc4-11eb-70-af-xxxx user=vpxuser:com.vmware.vsan.health] Need to retry fetchVsanSharedSecret. Count 3, Period 10, loginSeq 11303
<YYYY-MM-DD>T<TIME>Z info hostd[31281B70] [Originator@6876 sub=VsanSimsStubImpl opID=xxxxxxx-edc4-11eb-70-af-xxxx user=vpxuser:com.vmware.vsan.health] Need to retry fetchVsanSharedSecret. Count 2, Period 10, loginSeq 11304
<YYYY-MM-DD>T<TIME>Z info hostd[312C2B70] [Originator@6876 sub=VsanSimsStubImpl opID=xxxxxxx-edc4-11eb-70-af-xxxx user=vpxuser:com.vmware.vsan.health] Need to retry fetchVsanSharedSecret. Count 1, Period 10, loginSeq 11305
<YYYY-MM-DD>T<TIME>Z info hostd[31281B70] [Originator@6876 sub=VsanSimsStubImpl opID=xxxxxxx-edc4-11eb-70-af-xxxx user=vpxuser:com.vmware.vsan.health] Need to
<YYYY-MM-DD>T<TIME>Z error hostd[31A44B70] [Originator@6876 sub=Default opID=xxxxxxx-edc4-11eb-70-af-xxxx user=vpxuser:com.vmware.vsan.health] AdapterServer caught exception: SSL Exception: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
<YYYY-MM-DD>T<TIME>Z error hostd[31A44B70] [Originator@6876 sub=Default opID=xxxxxxx-edc4-11eb-70-af-xxxx user=vpxuser:com.vmware.vsan.health] Backtrace:

In the /var/log/vsanvpd.log file in the ESXi host, you see entries similar to:
<YYYY-MM-DD>T<TIME>Z vsanSoapServer: run:182:Failed to accept client <IP Address> [30]: SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
<YYYY-MM-DD>T<TIME>Z vsanSoapServer: run:186:SOAP process done
<YYYY-MM-DD>T<TIME>Z vsanSoapServer: run:139:To accept SOAP socket

In the /var/log/vmware/vsan-health/vmware-vsan-health-service.log located in the vCenter Server, you see entries similar to:
<YYYY-MM-DD>T<TIME>Z INFO vsan-health[sq1368:t2] [VsanMgmtAdapters::_HandleOneHost] Member info for host host-10(<ESXi hostname>) is (vim.cluster.VsanPerfMemberInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
thumbprint = '65374cbd9fe51889014158b834b6ef7be56e0fa7',
memberUuid = u'host-xxxxxxxxx-a9e9-d339-3642-xxxxxxxxx',
isSupportUnicast = true,
unicastAddressInfos = (vim.cluster.VsanUnicastAddressInfo)