Investigating disk space on an ESX or ESXi host
search cancel

Investigating disk space on an ESX or ESXi host

book

Article ID: 318926

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Virtual machine fails to power on
  • One or more of these errors might be seen in the VMware Infrastructure/vSphere Client:
    • Could not power on VM: No space left on device. Failed to power on VM
    • An unexpected error was received from the ESX host while powering on vm-##
       
  • The management agent (hostd) cannot start because the root partition is full
  • The vCenter Server agent (vpxa) cannot start because the root partition is full
  • The system cannot create any new files or directories on / or /tmp partition
  • The error that shows is similar:
     
    The root filesystem's file table is full. As a result, the file /testdir/4344 could not be created by the application 'scp'.

    no space left on device
     
  • The /tmp directory is filled with cimclient_root* logs, potentially thousands (~600,000 or more)
  • The vpxa log may contain entries similar to:

    [YYYY-MM-DDTHH:MM:SS] 'Libs' 3076454304 warning] Cannot make directory /var/run/vmware/root/27591: No space left on device
    [YYYY-MM-DDTHH:MM:SS] 'App' 3076454304 error] Exception: Failed to initialize authd server
    [YYYY-MM-DDTHH:MM:SS] 'App' 3076454304 error] Backtrace:
    [00] eip 0x909dd92
    [01] eip 0x9043444
    [02] eip 0x907f975
    [03] eip 0x908024c
    [04] eip 0x9033d74
     
  • In hostd logs, there are entries similar to:
    • The virtual machine could not start. Make sure VMware ESX is installed correctly and you have rights to run the software and to access all directories it uses, including directories in which the software is installed, directories containing the virtual disk and configuration files, and directories for temporary files.
    • error code -46
       
  • vMotion fails at 10%
  • VMware High Availability (HA) configuration produces the error:

    Configuration of host IP address is inconsistent on host <hostname>: address resolved to <IP Address> and <IPAddress>
     
  • In the /var/log/vmware/esxupdate.log file, this error can appear during the ESX host upgrade:

    Encountered error FileIOError: The error data is: Filename - None Message - I/O Error (28) on file : [Errno 28] No space left on device Errno - 10 Description - Unable to create, write or read a file as expected.
     
  • When trying to clone a virtual machine or when trying to hot migrate (vMotion) the virtual machine to another host, it shows the error:

    A general system error occurred: Failed to create journal file providerFailed to open "/var/log/vmware/journal/1269032951.9" for write


Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 8.0.0

Cause

These issues can occur if there is not enough free disk space on the ESX/ESXi host or if there is not enough space on the VMFS volume.
 
For troubleshooting purposes, checking the available free space should be done. This article provides steps to check the available disk space and steps to free space if required.

Resolution

Checking disk space usage on the ESX/ESXi service console partitions

To check the free space on an ESX/ESXi service console partitions:

  1. Open a console to the ESX/ESXi host. 
     
  2. Type df -h.

    For ESX, output is similar to:

    Filesystem Size Used Avail Use% Mounted on
    /dev/sda2 4.9G 3.0G 1.6G 66% /
    /dev/sda1 99M 18M 77M 19% /boot
    none 145M 0 145M 0% /dev/shm
    /dev/sda7 2.0G 135M 1.7G 8% /var/log
    [root@server]#

    Note: The partitions shown are from a default installation of an ESX host. If the partition configuration is modified, the output may appear differently.

    For ESXi,  output similar to:

    Filesystem Size Used Available Use% Mounted on

    visorfs 1.3G 322.3M 1.0G 24% /
    vmfs3 63.3G 570.0M 62.7G 1% /vmfs/volumes/4d71190d-########-####-########16ba
    vfat 285.9M 135.5M 150.4M 47% /vmfs/volumes/3c3693e8-########-####-########6d3a
    vfat 4.0G 2.7M 4.0G 0% /vmfs/volumes/4d71190d-########-####-########916ba
    vfat 249.7M 102.0M 147.7M 41% /vmfs/volumes/474ef17b-########-####-########916e
    vfat 249.7M 4.0k 249.7M 0% /vmfs/volumes/51aa187c-########-####-########c73c

  3. Review the Use% for each of the listed items. If any of the volumes listed are 100% full, they must be investigated to determine if space can be freed. The most important mount points to investigate on a default installation of ESX are the / and /var/log mounts because if they are full they can prevent proper operation of the ESX host.

  4. When finishing the review of output, type logout and press Enter to exit the system.

Checking disk space usage on a VMFS volume of an ESX/ESXi host

To check the free space on a VMFS volume of an ESX/ESXi host:

  1. Open a console to the ESX/ESXi host.
     
  2. Determine the free disk space on each filesystem using the command:

    vdf -h

    The output appears similar to:

    Filesystem Size Used Avail Use% Mounted on
    /dev/sda2 4.9G 3.0G 1.6G 66% /
    /dev/sda1 99M 18M 77M 19% /boot
    none 145M 0 145M 0% /dev/shm
    /dev/sda7 2.0G 135M 1.7G 8% /var/log
    /vmfs/devices 439G 0 439G 0% /vmfs/devices
    /vmfs/volumes/458865ba-########-####-#######e616
    68G 47G 20G 69% /vmfs/volumes/San_Storage
    /vmfs/volumes/45b5eb1a-########-####-#######5854b
    26G 9.7G 16G 36% /vmfs/volumes/Local_Storage

    Note: The partitions shown are dependent on the VMFS volumes that are defined and presented to the ESX or ESXi host.
     
  3. Review the Use% for each of the listed items. If any of the volumes listed are 100% full, they must be investigated to determine if space can be freed. If a VMFS volume is full,  new virtual machines cannot be created and any virtual machines that are using snapshots may fail.
     
  4. When the review of the output is finished, type logout and press Enter to exit the system.
Notes:
  • Another useful command is du -h --max-depth=1 <dir>. This command lists the directories within a given filesystem that contain the largest files. By starting at the root (/) directory and finding the largest directories,  drill down into these directories (using cd) and execute the same command recursively until finding the files themselves which are occupying space.
  • du -h is not available on ESXi 4.x. Reference the find commands in Identifying large files on an ESX or ESXi host to list the largest files.

</dir>

For related information, see Troubleshooting ESXi datastore or VMFS volume that is full or near capacity.

Identifying large files on an ESX or ESXi host

Information may accumulate on disks for several reasons. Log files may grow after a substantial amount of messages are written to them. Content such as virtual machines or ISOs may be copied to the ESX or ESXi host, but placed in an inappropriate location. Coredumps from past outages may have accumulated.

To confirm this, review the size of these directories:

  • The /vmimages/ directory is used to store operating system install files such as the VMware Tools or other ISO files.
  • The /var/core/ and /root/ directories are used to store crash files for on the service console and the VMkernel.
  • The /var/log/ directory is used to store the majority of the logs for the ESX host.
  • The /vmfs/volumes/ Datastores are used to store the virtual machine data.

To review the space consumed by several of these common directories, run this command:

du -ch /vmimages /var/core /root /var/log
 
 
Note: This command is not available on ESXi 4.x and ESXi 5.0.

If the reason for disk consumption can't be determined, use the find command to locate all files matching a given criteria.
For example, to find files within / that are larger than 10MB without traversing mount points, use the command:
find / -size +10M -exec du -h {} \; | less

To find files within /var/ that are larger than 1MB without traversing mount points, use the command:
 
find /var/ -size +1M -mount -exec du -h {} \; | less

Notes:
  • The -mount option to the find command is not available in ESXi.
  • The parameters for ESXi 5.0, 5.1 and 5.5 are different. For more information, see below.
 
-size N[bck] File size is N (c:bytes,k:kbytes,b:512 bytes(def.)).
+/-N: file size is bigger/smaller than N
-print Print (default and assumed)

Run the following for anything over 1M :

find / -size +1000k -exec du -h {} \;

Deleting unnecessary files

The following are a list of files that are safe to delete:

  • Old vm-support log bundles.
  • Virtual machines that are not being used and are not needed.
  • Old log files that are no longer needed.
  • Virtual machine log files can be removed if there are a large number (thousands) of vmware*.log files in the virtual machine's folder. 
  • ISO files that were copied to the system.

    Caution: Do not delete the VMware Tools ISO files from /vmimages/.

After determining what is consuming disk space, delete the unnecessary files:

  1. Open a console to the ESX or ESXi host.
     
  2. Use the rm command to permanently delete files. For example:

    rm /var/log/oldlogfile

    Caution: If a file is deleted, there is no way to recover it. Therefore, use caution when deleting files. If there is any hesitancy about deleting a specific file, contact VMware Support for assistance. If a system file is removed inadvertently, it may cause damage to the ESX or ESXi host that can require a re-installation of the software.

 

Zero out a log file

Should the files discovered with the du command be large and they don't need to be deleted, zero out the contents of a file with the standard redirect command.

Example: If file.log is 45MB, to keep the file and clear all the data from the file to reduce its size, run the following command from an SSH connection to the host:

 

# > file.log

The standard redirect of zero input into the file will cause the file to become 0kb in size and retain its name and permissions.

Archiving old files

Historical log files on an ESX or ESXi host may be required for reference in troubleshooting or trending. They can be compressed and archived rather than being deleted. Note, see also vmsupport files left on ESX or ESXi host fill the filesystem on which they reside.

To compress historic log files:

  1. Open a console to the ESX or ESXi host. 
     
  2. Compress the old log files.

    In ESX, run these commands to compress the old var/log/vmkwarning and /var/log/vmkernel log files:

    tar czvf /tmp/vmkwarning-logs.tgz /var/log/vmkwarning*
    tar czvf /tmp/vmkernel-logs.tgz /var/log/vmkernel.*
     
  3. In ESX and ESXi, run this command to compress the old /var/log/messages log files:

    tar czvf /tmp/messages-logs.tgz /var/log/messages.*
     
  4. Remove the source files using the command:

    rm /var/log/vmkwarning.* /var/log/vmkernel.* /var/log/messages.*
     
  5. Move the new archive files back to /var/log/ partition for long-term storage using the command:

    mv /tmp/vmkwarning-logs.tgz/tmp/vmkernel-logs.tgz/tmp/messages-logs.tgz /var/log/



Additional Information

Additional find command:

This command is for ESXi 4.x

find / -size +10240000c -exec vmkvsitools visorfsdu -h {} \; | less

For ESXi 5.x and later:

find / -size +10240000c -exec vmkvsitools vdu -h {} \; | less

Powered by Wolken Software