ESXi root ramdisk (/) reaches 100% capacity due to multiple auto-backup.sh scripts

Products

• VMware vSphere ESXi

Issue/Introduction

• The ESXi root ramdisk (/) reaches 100% capacity.
• Multiple auto-backup.sh scripts run on an hourly basis.
• Directories such as /tmp/auto-backup.######## fill the ramdisk.
• You observe errors similar to the following in the syslog.log file:

[DATE] In(78) crond[##########]: USER root pid ########## cmd /sbin/auto-backup.sh ++group=host/vim/vmvisor/auto-backup.sh
[DATE] In(14) auto-backup.sh[##########]: ConfigStore has been modified since the last backup
[DATE] Er(11) auto-backup.sh[##########]: failed to regenerate system configuration
[DATE] Er(11) auto-backup.sh[##########]: Failed command: crypto-util ++coreDumpEnabled=false,mem=15 envelope extract --aad ESXConfiguration /tmp/auto-backup.##########/local.tgz.ve /tmp/auto-backup.##########/local.tgz
[DATE] Er(11) auto-backup.sh[##########]: Error:
[DATE] Er(11) auto-backup.sh[##########]: Unable to decrypt encrypted system configuration

Environment

• VMware ESXi 8.0U3

Cause

A deeper system fault or configuration change causes the cron scheduler (crond) to spawn duplicate processes. This results in the auto-backup.sh script (and other hourly tasks) executing multiple times simultaneously, filling the ramdisk with temporary files and lock files. This situation stems from a combination of files being locked due to third-party applications, host performance, and the ramdisk filling up. When the ramdisk is full, crypto-util is unable to run, and multiple crond processes continue to run concurrently.

Resolution

Clear the auto-backup files and reboot the host to reset the cron scheduler.

Prerequisites:

• Identify the affected hosts by running vdu -ah / to confirm the root partition is full.

• Check for multiple active cron processes using ps -c | grep crond.

Steps:

1. Connect to the affected ESXi host via SSH and run the following command to clear existing files:

   • rm -rf /tmp/auto-backup.*

2. Place the host in Maintenance Mode.

3. Reboot the host.

Additional Information

Workaround: If you cannot immediately perform a permanent resolution (reboot), apply this workaround:

1. Connect to the affected ESXi host via SSH and run:

   • rm -rf /tmp/auto-backup.*

2. Locate the crond processes by running:

   • ps -c | grep crond

3. Using the PID obtained in the previous step, halt the crond processes:

   • kill <PID> <PID> <PID>

Subscribe to this knowledge article to get updates on this issue.