"The ramdisk 'root' is full" - ESXi Host Unable to Write Log Files

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts may experience errors indicating "The ramdisk 'root' is full" in the logs. When this occurs, log files such as syslog.log and vmkwarning.log cannot be written. The ramdisk usage percentage increases rapidly over time, potentially reaching critical levels within a few days. This issue prevents proper system logging and may impact troubleshooting capabilities. A reboot of the host temporarily improves the ramdisk space but it re-fills up and causes problems again at about the same rate as before.

Environment

• ESXi 8.0 and newer
• Host configuration where logging was originally designated for a location outside the ramdisk
• Logging becomes directed to a ramdisk location

Cause

ESXi 8.0 sets disk configurations differently depending on the available environment. This varies much more than previous ESXi versions did.

When ESXi is configured to store logs on external persistent storage, the system allocates a smaller amount of space for the ramdisk 'root' partition, as it's not expected to handle continuous logging operations. This smaller ramdisk is sufficient for temporary system operations but will quickly fill up if logging is inadvertently redirected back to the ramdisk.

This redirection can happen during configuration changes, or due to incorrect scratch location settings. The ramdisk, being memory-based with limited capacity, is not designed to store accumulated log files. When logs are written to this limited space instead of their intended persistent storage location, the ramdisk usage percentage increases rapidly, often reaching critical levels within a few days, resulting in the "The ramdisk 'root' is full" error.

Resolution

To resolve the "ramdisk 'root' is full" error:

1. Check the current ramdisk usage:

   1. Log in to the ESXi host using SSH

   2. Run the command vdf -h to view ramdisk usage

2. Verify where logs are being written:

   1. cd /scratch/log
      
      Note whether the resulting path is a persistent storage device.
      If not then this is being directed to the ramdisk
      If so then then logging to ramdisk is not your issue. Please follow ESXi host RAM disk is full to find what the issue is.

3. Follow System logs are stored on non-persistent storage to redirect logs to a persistent volume. These changes take place immediately.

4. Reboot the ESXi host to clear the ramdisk of the old logging

Additional Information

• The ramdisk is cleared during host reboots. While logging configuration changes take effect immediately, existing log files remain in the ramdisk until a reboot occurs.
• Monitor ramdisk usage periodically using the vdf -h command after making changes to ensure the resolution is effective.