Alarm 'Host error' on <ESXi_Host> triggered by event #### 'Issue detected on <ESXi_Host> in ha-datacenter: vmsyslog logger <Syslog_Server>:514 lost #### log messages'

Products

• VMware vSphere ESXi

Issue/Introduction

• This alarm is frequently triggered from the vCenter Server on an ESXi host.

Alarm 'Host error' on <ESXi_Host> triggered by event #### 'Issue detected on <ESXi_Host> in ha-datacenter: vmsyslog logger <Syslog_Server>:514 lost #### log messages'

• Test connectivity from the ESXi host to syslog server is successful:

nc -z <syslog_server> 514

• Attempting to open log files under /var/log on the ESXi host results in Input/Output error.
• The file /var/log/.vmsyslogd.err reports “failed to write log” and "failed to create file logger" messages, indicating issues with the /scratch partition:

YYYY-MM-DDT06:11:06.668Z vmsyslog.loggers.file    : ERROR   ] <hbr-agent> failed to write log, disabling
YYYY-MM-DDT06:11:06.671Z vmsyslog.loggers.file    : ERROR   ] Failed to create file logger </scratch/log/dellosname.log>

• When running df -h from the ESXi CLI, the datastore hosting the /scratch partition does not appear. Instead, the following errors are returned:

[root@esxi:~] df -h
VmFileSystem: Slow refresh failed: Cannot open volume /vmfs/volumes/########-########-####-############
VmFileSystem: Slow refresh failed: Unable to get FS Attrs for /vmfs/volumes/########-########-####-############
Error when running esxcli, return status was: 1
Errors:
Error getting data for filesystem on '/vmfs/volumes/########-########-####-############': Cannot open volume /vmfs/volumes/########-########-####-############, skipping.
Error getting data for filesystem on '/vmfs/volumes/########-########-####-############': Unable to get FS Attrs for /vmfs/volumes/########-########-####-############, skipping.

Where "########-########-####-############" is the UUID of the concerned datastore where /scratch partition resides.

• Attempts to browse the datastore via both vSphere Web Client and ESXi Host Client were unsuccessful.

Environment

VMware vSphere ESXi 8.x

VMware vCenter Server 8.x

Cause

• The backing device of the datastore appear as "not consumed" due to temporary inconsistencies such as memory glitches on the ESXi host, transient network disruptions, or file system irregularities. 

• The underlying disk backing the /scratch partition may be faulty.

Resolution

• Perform a storage rescan on the ESXi host (refer to the relevant KB for detailed steps). If the rescan does not resolve the issue, reboot the ESXi host.
• Engage the hardware vendor to investigate potential disk failure on the affected datastore.