ESXi host logging fails with "Logging to storage has failed" error when using a shared log directory

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms: 

• Host logging stops on multiple ESXi hosts, and logs are no longer being stored locally.
• Log file.vmsyslogd.err is created in /var/log directory
• Host services may be affected and fail, such as VM cloning operations.

The following error message is displayed in the affected hosts logs

/var/log/.vmsyslogd.err :

Logging to storage has failed. Logs are no longer being stored locally on this host 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

vSphere ESXi 

Cause

The root cause of the logging failure is a conflict in the logging directory. This conflict occurs because the ESXi host's default advanced setting, which controls the creation of unique log subfolders, is set to false and the path provided is the same path as other hosts. 

The setting causing the conflict is: Syslog.global.logDirUnique: false 

When this setting is false, and the directory path is the same on multiple hosts. The hosts do not create a unique subfolder per host in the logging directory, leading to issues when multiple hosts attempt to write their logs to the same location.

Resolution

This is a condition that may occur in a VMware vSphere environment.

To resolve this issue, enable the automatic creation of unique subfolders for the syslog directory by changing the advanced setting Syslog.global.logDirUnique to true. This configuration ensures each host creates a distinct, host-named subfolder in the logging directory, resolving the write conflict.

Follow the below steps: 

1. Access the ESXi host's Advanced Settings via the vSphere Client or command line.
2. Navigate to the Syslog.global configuration settings.
3. Change the value of Syslog.global.logDirUnique to true.
4. Reboot the ESXi host for the change to take effect.
5. Once the host reboots, it will create a unique subfolder within the main logging folder, and local logging will resume without conflict.

Additional Information

If you are unable to perform the configuration change or if the issue persists after following the resolution steps, please open a support case with Broadcom Support and refer to this KB article.