Configuring Scratch configuration and Syslog.global.auditRecord.storageEnable from Host Profile makes the esxi host 8.0U3 go unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

Changing scratch location when local audit recording is enabled and audit record storage is configured on scratch results in host going to "Not responding" state in vCenter

 

File /var/log/.vmsyslogd.err looks like:

2024-09-03T08:48:13.064Z vmsyslog                 : CRITICAL] vmsyslogd daemon starting (133622)
2024-09-03T08:48:13.455Z vmsyslog.loggers.audit   : ERROR   ] Files are missing from the audit record storage directory.

 

 

Environment

VMware ESXi 8.0.x

Cause

This issue is encountered when scratch is reconfigured to a different location on a host that has audit record storage enabled beforehand. 

1. When audit logging to local storage is enabled, the audit record storage directory is created containing the audit files, by default at /scratch/auditLog.
2. Scratch location may be reconfigured, which requires a host reboot to take effect.
3. When host comes up after reboot, the syslog daemon comes up and looks for the audit directory.
4. Since scratch partition now points to a different location, vmsyslogd is unable to find the audit directory and initialize audit record storage, causing it to throw an exception and crash.

 

 

 

Resolution

Engineering is aware of the issue and  working on a fix for this. The fix is expected to be included in future releases.

The issue can also be avoided by configuring audit logger after all the scratch configuration / host profiles are applied.

 

If the host has already arrived at the problematic state described above, to remediate:

Temporary workaround:

1. Disable local audit record storage (in advanced setting)
2. Enable local audit record storage
3. Check "/etc/init.d/vmsyslogd status". Start it if not running. 
4. Verify that logging is happening correctly.

5. Audit log files will look like: under path /vmfs/volumes/[datastore-name]/scratch/AuditLog

 

 

 

Additional Information

Some users may also encounter the "file table of the ramdisk "var" is full" error on vCenter. This might be a symptom of changing the log level of the syslog daemon from the default which results in vmsyslog.xxxxxxx.debug files to be created under /var/log/.

Please note that the syslog level should only be changed for debugging issues in vmsyslogd and should be set at the default "error" level under all normal circumstances as it may affect performance. These files may be deleted and the vmsyslogd log level reset to its default to avoid this particular symptom.