ESXi host not responding due to high disk space usage in the /tmp directory.

Products

• VMware vSphere ESXi

Issue/Introduction

• To provide information on how to troubleshoot and fix the /tmp partition becoming full on ESXi.
• In /var/run/log/vmkernel.log in ESXI you may see errors such as:

  YYYY-MM-DDTHH:MM:SS.323Z cpu16:75446085)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
  YYYY-MM-DDTHH:MM:SS.327Z cpu14:75334706)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/hp-mem.txt because its ramdisk (tmp) is full.
  YYYY-MM-DDTHH:MM:SS.877Z cpu3:44602881)<4>hpsa 0000:03:00.0: Device:C0:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.

   

• The /tmp folder shows 90% or higher usage. This is identified by running the vdf | grep tmp command.

  vdf | grep tmp

  Sample output

  tmp 192M 192M 0B 100% --

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x

Cause

Inadequate file cleanup processes.

Resolution

To resolve the issue, clear any large files from /tmp directory in ESXI host

Steps to follow:

1. SSH into the ESXi host with root.
2. Check if the /tmp directory is at 90% or more under the Use% column, run the following command:

   vdf | grep tmp

3. If the /tmp directory is at 90% or more under the Use% column by running the following commands:

   cd /tmp

   du -a <Path> | sort -n -r | head -n 20

4. Examine the directories for ones which are suspiciously large.
5. Within the suspicious directory, use the command to look at files.

   ls -lha

   If there are any files  that are very large, or which seem to have a very high number filling up the ramdisk.
   
   
6. Delete the unwanted files. Make sure these files are not being used before deleting.
7. Restart management services. Restarting Management Agents in ESXi

Additional Information

Impact/Risks:
If the directory fills up the ESXi host may fail to be able to update some files and may even go into a not-responding state.