Excessive log entries in /var/log/etcd.log

Products

• VMware vSphere ESXi

Issue/Introduction

• Excessive log entries in /var/log/etcd.log like below:
  
  YYYY-MM-DDTHH:MM:SSZ Er(x) etcd[xxxxxxx]: request sent was ignored (cluster ID mismatch: peer[xxxxxxxxxxxxxxxx]=xxxxxxxxxxxxxxxx, local=xxxxxxxxxxxxxxxx)
YYYY-MM-DDTHH:MM:SSZ Er(x) etcd[xxxxxxx]: request sent was ignored (cluster ID mismatch: peer[xxxxxxxxxxxxxxxx]=xxxxxxxxxxxxxxxx, local=xxxxxxxxxxxxxxxx)
YYYY-MM-DDTHH:MM:SSZ In(x) etcd[xxxxxxx]: rejected connection from "<remote ESXi IP>:<port number>" (error "remote error: tls: bad certificate", ServerName "")
YYYY-MM-DDTHH:MM:SSZ In(x) etcd[xxxxxxx]: rejected connection from "<remote ESXi IP>:<port number>" (error "remote error: tls: bad certificate", ServerName "")
  
  
• In /var/log/clusterAgent.log, there are entries like below:
  
  YYYY-MM-DDTHH:MM:SSZ No(x) clusterAgent[xxxxxxx]: WARN  grpc: addrConn.createTransport failed to connect to {<remote ESXi IP>::2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: write tcp <local ESXi IP>:<port number>-><remote ESXi IP>::2379: write: broken pipe". Reconnecting...
  
  
• Verify the connection from local ESXi to remote ESXi TCP port 2379 with command below, it can connect successully:
  
  openssl s_client -connect <remote ESXi IP>:2379

Environment

ESXi 8.0

Cause

ESXi host failed to connect to remote ESXi's TCP port 2379 due to "localhost" as "O" (Organization) in remote ESXi's certificate, which is incorrect.

Resolution

Renew remote ESXi host's certificate following the document below:

Renew or Refresh ESXi Certificates