Error: "The host ####:6514 has become unreachable" during forwarding to encrypted syslog server

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts report continuous below syslog-related alerts in host events and fail to forward logs to a remote encrypted syslog server. 
• The host "[syslog server IP]:6514" has become unreachable. Remote logging to this host has stopped
• The following errors are observed in the syslog server logs

[YYYY-MM-DD-THH:MM:SS] <syslog server name> rsyslogd[####]: not permitted to talk to peer '[Host IP]', certificate invalid: insecure algorithm [v8.2510.0-2.el9]
[YYYY-MM-DD-THH:MM:SS] <syslog server name> rsyslogd[####]: invalid cert info: peer provided 4 certificate(s). Certificate 1 info: certificate valid from [YYYY-MM-DD-THH:MM:SS] to [YYYY-MM-DD-THH:MM:SS]; Certificate public key: RSA; DN: C=US,ST=<State>,L=<City>,O=<Example Company>,OU=ABCR,CN=<ESXi hostname>.example.com,EMAIL=####@example.com; Issuer DN: DC=com,DC=example,DC=corp,CN=<Example Internal Issuing CA>; SAN:DNSname: <ESXi hostname>.example.com; [v8.2510.0-2.el9]

• In ESXi /var/run/log/vobd.log, the following entries are observed:

[YYYY-MM-DD-THH:MM:SS] In(14) vobd[####]: [UserLevelCorrelator] ####us: [esx.problem.vmsyslogd.remote.failure] The host "[syslog server IP]:6514" has become unreachable. Remote logging to this host has stopped.
[YYYY-MM-DD-THH:MM:SS] In(14) vobd[####]: The event ([esx.problem.vmsyslogd.remote.failure] The host "[syslog server IP]:6514" has become unreachable. Remote logging to this host has stopped.) was sent immediately to hostd;
[YYYY-MM-DD-THH:MM:SS] In(14) vobd[####]: [GenericCorrelator] ####us: [vob.user.vmsyslogd.remote.failure] The host "[syslog server IP]:6514" has become unreachable. Remote logging to this host has stopped.

• In ESXi /var/log/.vmsyslogd.err, the following entries are observed:
  
  [YYYY-MM-DD-THH:MM:SS] vmsyslog.loggers.network : ERROR ] Error shutting down socket.
[YYYY-MM-DD-THH:MM:SS] vmsyslog.loggers.network : ERROR ] [syslog server IP]:6514 - socket error : EOF occurred in violation of protocol (_ssl.c:2437)

Environment

VMware vSphere ESXi 8.x

Cause

The TLS connection failure to the encrypted syslog server is caused by an incomplete Subject Alternative Name (SAN) in the ESXi host certificate or an "insecure algorithm" rejection by modern syslog servers. Successful TLS connections require the certificate to contain the FQDN and IP address to satisfy peer validation.

Resolution

To resolve this issue and resume successful log forwarding, you must generate and apply a new certificate for the affected ESXi host with the complete Subject Alternative Name (SAN) entries.

1. Generate a new Certificate Signing Request (CSR) for the ESXi host. Refer Generating Certificate or Certificate Signing Request(CSR) 

2. Ensure the new CSR includes all of the following Subject Alternative Name (SAN) entries:

   • DNS (FQDN)

   • IP Address

3. Apply the newly generated certificate to the ESXi host.

4. If the ESXi host signing certificate does not match the syslog server, add the syslog signing certificate to the host. Follow the steps to configure syslog over SSL on ESXi to establish the trust.

5. Verify that the TLS connection to the syslog server is established.