ESXi/ESX error: No free space left on device

Products

• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• An ESXi/ESX host console indicates that/ is out of space.
• You see the error:
  
  No free space left on device
   
• A virtual machine fails to start.
• A log message reports that the file system is full.
• The df and vdf commands indicate that there is free space.
• You cannot disable VMware High Availability (HA).
• Processes become defunct.

Environment

VMware ESXi (all versions)

Resolution

If a filesystem has unused space, but the number of inodes has reached the maximum, you cannot create new files. This can appear like a full filesystem.

To resolve this issue, determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit and remove some files.

 

To determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit and remove some files:

1. Gain root access to the command line.
   • For ESXi, open a SSH session.
      
2. On the ESX host, run the command:
   
   [root@esx /]$ stat -f /
   
   You see output similar to:
   
   File: "/"
   ID: 0 Namelen: 255 Type: ext2/ext3
   Blocks: Total: 1259079 Free: 898253 Available: 834295 Size: 4096
   Inodes: Total: 640000 Free: 580065
   
   In this example, there are 640000 inodes total and 580065 are free, so inodes are available.
    
3. If, however, you have exhausted the inode limit, you must remove some files. For more information on freeing disk space, see:
    
   • Investigating disk space on an ESX or ESXi host (1003564)
   • Freeing ESXi inodes (1008643)
      
4. Run this command to generate a list of log files that may be candidates for deletion:
   
   find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;
   
   This command seeks out files 50,000 KB (50MB) or larger, skips the /vmfs directory where virtual machines may be located, and provides detailed output for each file. This may return locally-stored ISO files, log bundles, large log files, etc.
   
   Some typical paths where excess files may be located are:
   • /tmp/vmware-root
   • /var/core
   • /var/log/vmware/aam/rul
      
   Warning:
   • Review the files in question before removal using rm. Files cannot be recovered after deletion.
   • Do not delete log information unless necessary. Consider archiving older logging elsewhere.
      
5. Run this command against the VMFS volume:
   
   vmkfstools -P -v 10 /vmfs/volumes/########-####-########a748
   
   You see output similar to:
   
   VMFS-3.21 file system spanning 1 partitions.
   File system label (if any): lun1 (2)
   Mode: public
   Capacity 5100273664 (4864 file blocks * 1048576), 148897792 (142 blocks) avail
   Volume Creation Time: Tue Jun 12 12:08:41 2007
   Files (max/free): 30720/30707
   Ptr Blocks (max/free): 61440/61436
   Sub Blocks (max/free): 3968/3963
   UUID: ########-####-########a748
   Partitions spanned:
   vmhba1:0:1:1
   
   In this example, none of the values for Files, Ptr Blocks and Sub Blocks are at zero so there are available VMFS entries for new files.