nvmelogs causing /tmp folder to be filled up and host goes into not responding state

Products

• VMware vSphere ESXi

Issue/Introduction

 

Symptoms:

• An ESXi host shows Not Responding in vCenter Server
• ESXi hosts shows Not Responding in vCenter for 10-15 minutes then may come back to normal
• hostd.log may show:

YYYY-MM-DDTHH:MM:SS.012Z info hostd[2104181] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
-->    key = 95,
-->    chainId = -1,
-->    createdTime = "YYYY-MM-DDTHH:MM:SSZ",
-->    userName = "",
-->    host = (vim.event.HostEventArgument) {
-->       name = "ESX-0X-FQDN",
-->       host = 'vim.HostSystem:ha-host'
-->    },
-->    eventTypeId = "esx.problem.visorfs.ramdisk.full",
-->    arguments = (vmodl.KeyAnyValue) [
-->       (vmodl.KeyAnyValue) {
-->          key = "1",
-->          value = "tmp"
-->       },
-->       (vmodl.KeyAnyValue) {
-->          key = "2",
-->          value = "/tmp/nvmelogs/vmhba12_host_tele.bin"
--> }

 

• vmkernel.log may show:

YYYY-MM-DDTHH:MM:SS.821Z cpu5:9078600)Admission failure in path: host/system/visorfs/ramdisks/tmp:tmp
YYYY-MM-DDTHH:MM:SS.821Z cpu5:9078600)tmp (1370) extraMin/extraFromParent: 1/1, tmp (1369) childEmin/eMinLimit: 65536/65536
YYYY-MM-DDTHH:MM:SS.821Z cpu5:9078600)Admission failure in path: host/system/visorfs/ramdisks/tmp:tmp
YYYY-MM-DDTHH:MM:SS.821Z cpu5:9078600)tmp (1370) extraMin/extraFromParent: 1/1, tmp (1369) childEmin/eMinLimit: 65536/65536
YYYY-MM-DDTHH:MM:SS.821Z cpu5:9078600)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba12_host_tele.bin because its ramdisk (tmp) is full.

• vmkwarning.log may show:

YYYY-MM-DDTHH:MM:SS.687Z cpu32:9078841)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba23_host_tele.bin because its ramdisk (tmp) is full.
YYYY-MM-DDTHH:MM:SS.844Z cpu11:9078885)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba23_controller_tele.bin because its ramdisk (tmp) is full.
YYYY-MM-DDTHH:MM:SS.152Z cpu26:9078897)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba23_persistent_event.bin because its ramdisk (tmp) is full.
YYYY-MM-DDTHH:MM:SS.761Z cpu44:9078957)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba22_controller_tele.bin because its ramdisk (tmp) is full.

Environment

VMware vSphere ESXi 6.5.x
VMware vSphere ESXi 6.7.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The vm-support command will execute nvmelogs.sh to collect NVMe telemetry and persistent event logs. NVMe logs occupy a large space. After the vm-support collection is completed, they will be deleted.

The /tmp directory is a temporary folder to store the files temporarily. These files will ultimately be saved with the vm-support bundle.

Resolution

This issue is fixed in 7.0 U3q. Refer to VMware ESXi 7.0 Update 3q Release Notes for more details. 

Workaround for version VMware ESXi 7.0 Update 3q and below: 

• From CLI, execute the "vm-support -e Storage:NVMe" command instead of "vm-support" to disable nvmelogs when collecting a support bundle on the ESXi host.
  OR
• From the vCenter GUI, deselect NVMe manifest (under Storage) while generating a support bundle from vCenter.
  
  

 

Additional Information