ESXi/ESX error: No free space left on device
search cancel

ESXi/ESX error: No free space left on device

book

Article ID: 342658

calendar_today

Updated On:

Products

VMware vSAN VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • An ESXi/ESX host console indicates that/ is out of space.
  • You see the error:

    No free space left on device
     
  • A virtual machine fails to start.
  • A log message reports that the file system is full.
  • The df and vdf commands indicate that there is free space.
  • You cannot disable VMware High Availability (HA).
  • Processes become defunct.


Environment

VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.1
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.5
VMware ESX Server 3.0.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 5.0
VMware ESX 4.1.x
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 6.0

Resolution

If a filesystem has unused space, but the number of inodes has reached the maximum, you cannot create new files. This can appear like a full filesystem.

To resolve this issue, determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit and remove some files.
 
To determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit and remove some files:
  1. Gain root access to the command line.
    • For ESXi, open a SSH session.
       
  2. On the ESX host, run the command:

    [root@esx /]$ stat -f /

    You see output similar to:

    File: "/"
    ID: 0 Namelen: 255 Type: ext2/ext3
    Blocks: Total: 1259079 Free: 898253 Available: 834295 Size: 4096
    Inodes: Total: 640000 Free: 580065

    In this example, there are 640000 inodes total and 580065 are free, so inodes are available.
     
  3. If, however, you have exhausted the inode limit, you must remove some files. For more information on freeing disk space, see:
     
  4. Run this command to generate a list of log files that may be candidates for deletion:

    find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;

    This command seeks out files 50,000 KB (50MB) or larger, skips the /vmfs directory where virtual machines may be located, and provides detailed output for each file. This may return locally-stored ISO files, log bundles, large log files, etc.

    Some typical paths where excess files may be located are:
    • /tmp/vmware-root
    • /var/core
    • /var/log/vmware/aam/rul
       
    Warning:
    • Review the files in question before removal using rm. Files cannot be recovered after deletion.
    • Do not delete log information unless necessary. Consider archiving older logging elsewhere.
       
  5. Run this command against the VMFS volume:

    vmkfstools -P -v 10 /vmfs/volumes/466e7eb9-3692da25-a15c-001321caa748

    You see output similar to:

    VMFS-3.21 file system spanning 1 partitions.
    File system label (if any): lun1 (2)
    Mode: public
    Capacity 5100273664 (4864 file blocks * 1048576), 148897792 (142 blocks) avail
    Volume Creation Time: Tue Jun 12 12:08:41 2007
    Files (max/free): 30720/30707
    Ptr Blocks (max/free): 61440/61436
    Sub Blocks (max/free): 3968/3963
    UUID: 466e7eb9-3692da25-a15c-001321caa748
    Partitions spanned:
    vmhba1:0:1:1

    In this example, none of the values for Files, Ptr Blocks and Sub Blocks are at zero so there are available VMFS entries for new files.



Additional Information

Powered by Wolken Software