ESXi host not responding , /tmp folder shows high usage(90%-100%)

Products

• VMware vSphere ESXi

Issue/Introduction

To provide information on how to troubleshoot and fix the /tmp partition becoming full on ESXi.

Symptoms:

• In vmkernel.log, you may see errors such as:

2017-04-06T13:06:16.323Z cpu16:75446085)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
2017-04-06T13:06:16.327Z cpu14:75334706)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/hp-mem.txt because its ramdisk (tmp) is full.
2017-04-06T13:06:59.877Z cpu3:44602881)<4>hpsa 0000:03:00.0: Device:C0:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.

• The /tmp folder shows 90% or higher usage.

tmp 192M 192M 0B 100% --

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0.0

Cause

Inadequate file cleanup processes.

Resolution

Check whether the /tmp directory is full:

1. If the /tmp directory is at 90% or more under the Use% column, run the following commands:

vdf |grep tmp

2.  If the /tmp directory is at 90% or more under the Use% column, run the following commands:

cd /tmp
du -a <Path> | sort -n -r | head -n 20

3. Examine the directories for ones which are suspiciously large.
4. Within the suspicious directory, use the command to look at files.

ls -lha

Note: If there are any that are very large, or which seem to have a very high number filling up the ramdisk.

 

Large/frequent files name	Resolution	Product
mili2d.log	Delete the file Restart management services
file names similar to: proto_2019_09_01T16-29-56.log	/tmp on ESXi hosts configured for NSX-T may become heavily utilized or full	NSX-T
ahs.debug	The event log of HPE ProLiant hosts shows error: The ramdisk 'tmp' is full. As a result, the file /tmp/auto-backup.1619198/local.tgz could not be written
ql_ima.log ql-ima_sdm.log ql_ima_sdm.log.old	ESXi hosts are not responding as ql_ima.log fill up the /tmp directory
current.png.xxxx	Many current.png.xxxx on the /tmp on ESXi host causes hostd to crash and disconnect	vCloud Director

Additional Information

Impact/Risks:
If the directory fills up the ESXi host may fail to be able to update some files and may even go into a not-responding state.