Investigating disk space on an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

The following symptoms may be observed:

A virtual machine fails to power on
One or more of these errors may be seen in the vSphere Client:

Could not power on VM: No space left on device. Failed to power on VM

An unexpected error was received from the ESXi host while powering on vm-##

The management agent (hostd) cannot start because the root partition is full
The vCenter Server agent (vpxa) cannot start because the root partition is full
vMotion fails at 10%
The /tmp directory is filled with cimclient_root* logs (potentially thousands)
The system cannot create new files or directories on the / or /tmp partition
- The error observed may be similar to:
  
  The root filesystem's file table is full. As a result, the file /dir could not be created by the application 'scp'
  
  no space left on device
In the /var/run/log/vpxa.log file, the following entries may be observed:

[YYYY-MM-DDTHH:MM:SS] 'Libs' 3076454304 warning] Cannot make directory /var/run/vmware/root/27591: No space left on device
[YYYY-MM-DDTHH:MM:SS] 'App' 3076454304 error] Exception: Failed to initialize authd server
[YYYY-MM-DDTHH:MM:SS] 'App' 3076454304 error] Backtrace:
[00] eip 0x909dd92
[01] eip 0x9043444
[02] eip 0x907f975
In the /var/run/log/hostd.log file, the following entries may be observed:
- The virtual machine could not start. Make sure VMware ESXi is installed correctly and you have rights to run the software and to access all directories it uses, including directories in which the software is installed, directories containing the virtual disk and configuration files, and directories for temporary files.
- error code -46
VMware High Availability (HA) configuration may produce the following error:

Configuration of host IP address is inconsistent on host <hostname>: address resolved to <IP Address> and <IPAddress>
In the /var/log/vmware/esxupdate.log file, this error may be observed:

Encountered error FileIOError: The error data is: Filename - None Message - I/O Error (28) on file : [Errno 28] No space left on device Errno - 10 Description - Unable to create, write or read a file as expected.
When trying to clone a virtual machine or when trying to vMotion the virtual machine to another host, it may show the following error:

A general system error occurred: Failed to create journal file providerFailed to open "/var/log/vmware/journal/1269032951.9" for write

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

These issues may occur if there is not enough free disk space on an ESXi host or if there is not enough space on the VMFS volume.

Resolution

Checking disk space usage on ESXi

To check disk space usage of all currently mounted file systems on an ESXi host:

Open a console to the ESXi host. See Using ESXi Shell in ESXi
Determine disk space usage using the following command:# df -h
Filesystem Size Used Available Use% Mounted on

visorfs 1.3G 322.3M 1.0G 24% /
vmfs3 63.3G 570.0M 62.7G 1% /vmfs/volumes/4d71190d-########-####-########16ba
vfat 285.9M 135.5M 150.4M 47% /vmfs/volumes/3c3693e8-########-####-########6d3a
vfat 4.0G 2.7M 4.0G 0% /vmfs/volumes/4d71190d-########-####-########916ba
Review the Use% for each of the filesystems. If any of the volumes listed are 100% full, they must be inspected to determine if space can be freed up. The most important mount points to investigate on a default installation of ESXi are the / and /var/log mounts, as they can prevent proper operation of the ESXi host.

Checking Ramdisk space usage on a VMFS volume of an ESXi host

To check the Ramdisk space usage of an ESXi host:

Open a console to the ESXi host.
Determine Ramdisk usage using the following command:

# vdf -h

----- Ramdisk Size Used Available Use% Mounted on root 32M 6M 25M 20% -- etc 28M 2M 25M 7% -- opt 32M 0B 32M 0% -- var 48M 492K 47M 1% -- tmp 256M 0B 256M 0% -- iofilters 32M 0B 32M 0% -- shm 1024M 0B 1024M 0% -- crx 1024M 0B 1024M 0% -- configstore 32M 188K 31M 0% -- configstorebkp 32M 184K 31M 0% -- hostdstats 96M 3M 92M 3% --
Review the Use% column for each of the Ramdisks. If any are listed as 100% full, they must be investigated to determine if space can be freed up.

Identifying large files on an ESXi host

Information may accumulate on disks for several reasons:

Log files may grow after a substantial amount of messages are written to them.
Content such as virtual machines or ISOs may be copied to the ESXi host, but placed in an inappropriate location.
Coredumps from past outages may have accumulated.

To confirm this, review the size of these directories:

The /vmimages/ directory is used to store operating system install files such as the VMware Tools or other ISO files.
The /var/core/ and /root/ directories are used to store crash files for on the service console and the VMkernel.
The /var/log/ directory is used to store the majority of the logs for the ESXi host.
The /vmfs/volumes/ Datastores are used to store the virtual machine data.

To review the space consumed by several of these common directories, run this command:

# du -ch /vmimages /var/core /root /var/log

12.0K /vmimages
4.0K /var/core
8.0K /root
4.0K /var/log/ntp
4.0K /var/log/vmware/journal
8.0K /var/log/vmware
556.0K /var/log
580.0K total

If the reason for disk consumption can't be determined, use the find command to locate all files matching a given criteria.
To find the top 10 files within / that are larger than 100MB without traversing mount points, use the command:

# find / -size +100000k -exec du {} \; | sort -r |head -n 10

4193280 ./dev/disks/mpx.vmhba0:C0:T0:L0:6
4193280 ./dev/disks/mpx.vmhba0:C0:T0:L0:5
263168 ./vmfs/volumes/aabbccdd-aabbccdd-8e66-000c29483b7a/.jbc.sf
263168 ./vmfs/volumes/aabbccdd-aabbccdd-868f-000c29483b7a/.jbc.sf
252288 ./tardisks/sb.v00
245876 ./tardisks/vim.v00
209715200 ./dev/disks/mpx.vmhba0:C0:T1:L0
209713153 ./dev/disks/mpx.vmhba0:C0:T1:L0:1
209453056 ./dev/lvm/aabbccdd-aabbccdd-b5d6-000c29483b7a

Deleting unnecessary files

The following are a list of files that are safe to delete:

Old vm-support log bundles
Virtual machines that are not needed
Old log files that are no longer needed
Virtual machine log files can be removed if there are a large number (hundreds) of vmware#.log files in the virtual machine's folder.
ISO files that were copied to the system.

Caution: Do not delete the VMware Tools ISO files from /vmimages/

To delete the unnecessary files:

Open a console to the ESXi host.
Use the rm command to permanently delete files:

# rm /var/log/oldlogfile rm: remove '/var/log/oldlogfile'? Y
Caution: If a file is deleted, there is no way to recover it. Therefore, use caution when deleting files. If there is any hesitancy about deleting a specific file, contact Broadcom Technical Support for assistance. If a system file is removed inadvertently, it may cause damage to the ESXi host that can require a re-installation of the software.

Archiving old files

Historical log files on an ESXi host may be required for reference in troubleshooting. They can be compressed and archived rather than being deleted.

To compress historic log files:

Open a console to the ESXi host.
Compress the old var/log/vmkwarning and /var/log/vmkernel log files:

# tar czvf /tmp/vmkfiles.tgz var/log/vmkwarning.* var/log/vmkernel.*