Investigating disk space on an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

There may be the following scenarios:

A virtual machine fails to power on
One or more of these errors may be seen in the vSphere Client:

Could not power on VM: No space left on device. Failed to power on VM

An unexpected error was received from the ESXi host while powering on vm-##

The management agent (hostd) cannot start because the root partition is full
The vCenter Server agent (vpxa) cannot start because the root partition is full
The system cannot create new files or directories on the / or /tmp partition
The error that shows may be similar to:

The root filesystem's file table is full. As a result, the file /dir could not be created by the application 'scp'

no space left on device
The /tmp directory is filled with cimclient_root* logs, (potentially thousands)
The vpxa log may contain entries similar to:

[YYYY-MM-DDTHH:MM:SS] 'Libs' 3076454304 warning] Cannot make directory /var/run/vmware/root/27591: No space left on device
[YYYY-MM-DDTHH:MM:SS] 'App' 3076454304 error] Exception: Failed to initialize authd server
[YYYY-MM-DDTHH:MM:SS] 'App' 3076454304 error] Backtrace:
[00] eip 0x909dd92
[01] eip 0x9043444
[02] eip 0x907f975
In hostd logs, there may be entries similar to:
- The virtual machine could not start. Make sure VMware ESXi is installed correctly and you have rights to run the software and to access all directories it uses, including directories in which the software is installed, directories containing the virtual disk and configuration files, and directories for temporary files.
- error code -46
vMotion fails at 10%
VMware High Availability (HA) configuration may produce the error:

Configuration of host IP address is inconsistent on host <hostname>: address resolved to <IP Address> and <IPAddress>
In the /var/log/vmware/esxupdate.log file, this error may appear during the ESXi host upgrade:

Encountered error FileIOError: The error data is: Filename - None Message - I/O Error (28) on file : [Errno 28] No space left on device Errno - 10 Description - Unable to create, write or read a file as expected.
When trying to clone a virtual machine or when trying to hot migrate (vMotion) the virtual machine to another host, it may show the error:

A general system error occurred: Failed to create journal file providerFailed to open "/var/log/vmware/journal/1269032951.9" for write

Environment

VMware vSphere ESXi 6.5.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

These issues may occur if there is not enough free disk space on the ESXi host or if there is not enough space on the VMFS volume.

Resolution

Checking disk space usage on ESXi

To check the free space on an ESXi service console partitions:

Open a console to the ESXi host.
Determine the free disk space# df -h
Filesystem Size Used Available Use% Mounted on

visorfs 1.3G 322.3M 1.0G 24% /
vmfs3 63.3G 570.0M 62.7G 1% /vmfs/volumes/4d71190d-########-####-########16ba
vfat 285.9M 135.5M 150.4M 47% /vmfs/volumes/3c3693e8-########-####-########6d3a
vfat 4.0G 2.7M 4.0G 0% /vmfs/volumes/4d71190d-########-####-########916ba
Review the Use% for each of the listed items. If any of the volumes listed are 100% full, they must be investigated to determine if space can be freed. The most important mount points to investigate on a default installation of ESXi are the / and /var/log mounts, as they can prevent proper operation of the ESXi host.

Checking Ramdisk space usage on a VMFS volume of an ESXi host

To check the Ramdisk space on a VMFS volume of an ESXi host:

Open a console to the ESXi host.
Determine the free disk space on each filesystem:

# vdf -h

----- Ramdisk Size Used Available Use% Mounted on root 32M 6M 25M 20% -- etc 28M 2M 25M 7% -- opt 32M 0B 32M 0% -- var 48M 492K 47M 1% -- tmp 256M 0B 256M 0% -- iofilters 32M 0B 32M 0% -- shm 1024M 0B 1024M 0% -- crx 1024M 0B 1024M 0% -- configstore 32M 188K 31M 0% -- configstorebkp 32M 184K 31M 0% -- hostdstats 96M 3M 92M 3% --
Review the Use% column for each of the listed items. If any of the volumes listed are 100% full, they must be investigated to determine if space can be freed.

Identifying large files on an ESXi host

Information may accumulate on disks for several reasons. Log files may grow after a substantial amount of messages are written to them. Content such as virtual machines or ISOs may be copied to the ESXi host, but placed in an inappropriate location. Coredumps from past outages may have accumulated.

To confirm this, review the size of these directories:

The /vmimages/ directory is used to store operating system install files such as the VMware Tools or other ISO files.
The /var/core/ and /root/ directories are used to store crash files for on the service console and the VMkernel.
The /var/log/ directory is used to store the majority of the logs for the ESXi host.
The /vmfs/volumes/ Datastores are used to store the virtual machine data.

To review the space consumed by several of these common directories, run this command:

# du -ch /vmimages /var/core /root /var/log

12.0K /vmimages
4.0K /var/core
8.0K /root
4.0K /var/log/ntp
4.0K /var/log/vmware/journal
8.0K /var/log/vmware
556.0K /var/log
580.0K total

If the reason for disk consumption can't be determined, use the find command to locate all files matching a given criteria.
To find the top 10 files within / that are larger than 100MB without traversing mount points, use the command:

# find / -size +100000k -exec du {} \; | sort -r |head -n 10

4193280 ./dev/disks/mpx.vmhba0:C0:T0:L0:6
4193280 ./dev/disks/mpx.vmhba0:C0:T0:L0:5
263168 ./vmfs/volumes/aabbccdd-aabbccdd-8e66-000c29483b7a/.jbc.sf
263168 ./vmfs/volumes/aabbccdd-aabbccdd-868f-000c29483b7a/.jbc.sf
252288 ./tardisks/sb.v00
245876 ./tardisks/vim.v00
209715200 ./dev/disks/mpx.vmhba0:C0:T1:L0
209713153 ./dev/disks/mpx.vmhba0:C0:T1:L0:1
209453056 ./dev/lvm/aabbccdd-aabbccdd-b5d6-000c29483b7a

Deleting unnecessary files

The following are a list of files that are safe to delete:

Old vm-support log bundles
Virtual machines that are not needed
Old log files that are no longer needed
Virtual machine log files can be removed if there are a large number (hundreds) of vmware#.log files in the virtual machine's folder.
ISO files that were copied to the system.

Caution: Do not delete the VMware Tools ISO files from /vmimages/

To delete the unnecessary files:

Open a console to the ESXi host.
Use the rm command to permanently delete files:

# rm /var/log/oldlogfile rm: remove '/var/log/oldlogfile'? Y
Caution: If a file is deleted, there is no way to recover it. Therefore, use caution when deleting files. If there is any hesitancy about deleting a specific file, contact Broadcom Technical Support for assistance. If a system file is removed inadvertently, it may cause damage to the ESXi host that can require a re-installation of the software.

Archiving old files

Historical log files on an ESXi host may be required for reference in troubleshooting. They can be compressed and archived rather than being deleted.

To compress historic log files:

Open a console to the ESXi host.
Compress the old var/log/vmkwarning and /var/log/vmkernel log files:

# tar czvf /tmp/vmkfiles.tgz var/log/vmkwarning.* var/log/vmkernel.*