Configuring Scratch configuration and Syslog.global.auditRecord.storageEnable causes the esxi host to go unresponsive or hostd not logging

Products

• VMware vSphere ESXi

Issue/Introduction

Changing scratch location when local audit recording is enabled and audit record storage is configured on scratch results in host going to "Not responding" state in vCenter

 

File /var/log/.vmsyslogd.err looks like:

[YYYY-MM-DDTHH:MM:SS] vmsyslog                 : CRITICAL] vmsyslogd daemon starting (133622)
[YYYY-MM-DDTHH:MM:SS] vmsyslog.loggers.audit   : ERROR   ] Files are missing from the audit record storage directory.

 

Alternatively this could cause the hostd process to stop updating hostd.log

Environment

VMware ESXi 8.0.x

VMware ESXi 7.0.x

Cause

This issue is encountered when scratch is reconfigured to a different location on a host that has audit record storage enabled beforehand. 

1. When audit logging to local storage is enabled, the audit record storage directory is created containing the audit files, by default at /scratch/auditLog.
2. Scratch location may be reconfigured, which requires a host reboot to take effect.
3. When host comes up after reboot, the syslog daemon comes up and looks for the audit directory.
4. Since scratch partition now points to a different location, vmsyslogd is unable to find the audit directory and initialize audit record storage, causing it to throw an exception and crash.

 

 

 

Resolution

Engineering is aware of the issue and  working on a fix for this. The fix is expected to be included in future releases.

The issue can also be avoided by configuring audit logger after all the scratch configuration / host profiles are applied.

 

If the host has already arrived at the problematic state described above, to remediate:

Temporary workaround:

1. Disable local audit record storage (in advanced setting)

2. Enable local audit record storage

Alternatively can disable and enable the local audit record storage from a CLI to the esxi host with the following commands:

     esxcli system auditrecords local disable
     esxcli system auditrecords local enable

3. Check  vmsyslogd status with command: /etc/init.d/vmsyslogd status.  If it is not running start it with the command:  /etc/init.d/vmsyslogd start

4. Verify that logging is happening correctly.

 

5. Audit log files will look like: /vmfs/volumes/[datastore-name]/scratch/AuditLog

 

 

 

Additional Information

Some users may also encounter the "file table of the ramdisk "var" is full" error on vCenter. This might be a symptom of changing the log level of the syslog daemon from the default which results in vmsyslog.xxxxxxx.debug files to be created under /var/log/.

Please note that the syslog level should only be changed for debugging issues in vmsyslogd and should be set at the default "error" level under all normal circumstances as it may affect performance. These files may be deleted and the vmsyslogd log level reset to its default to avoid this particular symptom.