Constant "VFAT volume is full" warnings in the log for a volume that is not full

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Constant warnings are reported in the logs regarding VFAT volume being full.
• Alerts from VMware Log Insight reports message similar to:
  
  " A root file system has reached its resource pool limit. various administrative actions depend on the ability to write files to various parts of the root file system and might fail if the ram disk and/or inode table is full"
   
• In the vmkwarning.log file, you see error similar to:
  
  2016-11-10T21:10:01.459Z cpu27:3864160)WARNING: VFAT: 313: VFAT volume naa.614187704eb170001e099eb213286bc2:2 (UUID 57df8c88-7c0644d1-59e0-1c4024c3cd9e) is full. (8386560 sectors, 7424128 free sectors)

• In the vobd.log file, you see error similar to:
  
  2016-11-10T20:48:01.408Z: [VfatCorrelator] 1163731001101us: [vob.vfat.filesystem.full] VFAT volume naa.614187704eb170001e099eb213286bc2:2 (UUID 57df8c88-7c0644d1-59e0-1c4024c3cd9e) is full. (8386560 sectors, 7426688 free sectors)

• In the vmkernel.log file, you see error similar to:
  
  WARNING: VFAT: 313: VFAT volume (UUID 57df8c88-7c0644d1-59e0-1c4024c3cd9e ) is full.

• df -h on /vmfs/volumes shows enough free space left on vfat partition.
• ls -lR output on the volume shows vsantraces and observer files.

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Courier; color: #323333; -webkit-text-stroke: #323333} span.s1 {font-kerning: none; text-shadow: 0.0px 1.0px 0.0px #000000}

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0

Cause

There is an artificial limit on the number of files with the same prefix.

Resolution

This issue is resolved in VMware ESXi 6.0 Update 3, available at VMware Download

 

To workaround this issue if you do not want to upgrade:

 

1. Confirm that there are >1000 files in the vsantraces.
2. Delete the vSAN trace/observer files from the /vmfs/volumes/<UUID>/vsantraces location.
   
   If it is not vSAN, contact VMware technical support.