nvmelogs causing /tmp folder to be filled up and host goes to not responding state.

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

An ESXi host is not responding in the vCenter Server inventory.
ESXi hosts goes to "not responding" in vCenter for 10-15 minutes and comes back normal.
hostd.log:

2022-08-02T14:43:08.012Z info hostd[2104181] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

--> key = 95,
--> chainId = -1,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> host = (vim.event.HostEventArgument) {
--> name = "glri1-vcf-wld1-cl2-esx5.cmcom.cloud",
--> host = 'vim.HostSystem:ha-host'
--> },
--> eventTypeId = "esx.problem.visorfs.ramdisk.full",
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "tmp"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "2",
--> value = "/tmp/nvmelogs/vmhba12_host_tele.bin"
--> }
--> ],
--> objectId = "ha-host",
--> objectType = "vim.HostSystem",
--> }

vmkernel.log:

2022-08-03T07:27:38.821Z cpu5:9078600)Admission failure in path: host/system/visorfs/ramdisks/tmp:tmp
2022-08-03T07:27:38.821Z cpu5:9078600)tmp (1370) extraMin/extraFromParent: 1/1, tmp (1369) childEmin/eMinLimit: 65536/65536
2022-08-03T07:27:38.821Z cpu5:9078600)Admission failure in path: host/system/visorfs/ramdisks/tmp:tmp
2022-08-03T07:27:38.821Z cpu5:9078600)tmp (1370) extraMin/extraFromParent: 1/1, tmp (1369) childEmin/eMinLimit: 65536/65536
2022-08-03T07:27:38.821Z cpu5:9078600)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba12_host_tele.bin because its ramdisk (tmp) is full.
2022-08-03T07:27:38.979Z cpu0:9078650)Admission failure in path: host/system/visorfs/ramdisks/tmp:tmp
2022-08-03T07:27:38.979Z cpu0:9078650)tmp (1370) extraMin/extraFromParent: 1/1, tmp (1369) childEmin/eMinLimit: 65536/65536
2022-08-03T07:27:38.979Z cpu0:9078650)Admission failure in path: host/system/visorfs/ramdisks/tmp:tmp
2022-08-03T07:27:38.979Z cpu0:9078650)tmp (1370) extraMin/extraFromParent: 1/1, tmp (1369) childEmin/eMinLimit: 65536/65536
2022-08-03T07:27:38.979Z cpu0:9078650)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba12_controller_tele.bin because its ramdisk (tmp) is full.

vmkwarning.log:

022-08-03T07:27:44.687Z cpu32:9078841)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba23_host_tele.bin because its ramdisk (tmp) is full.
2022-08-03T07:27:44.844Z cpu11:9078885)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba23_controller_tele.bin because its ramdisk (tmp) is full.
2022-08-03T07:27:45.152Z cpu26:9078897)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba23_persistent_event.bin because its ramdisk (tmp) is full.
2022-08-03T07:27:46.604Z cpu40:9078910)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba22_host_tele.bin because its ramdisk (tmp) is full.
2022-08-03T07:27:46.761Z cpu44:9078957)WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/nvmelogs/vmhba22_controller_tele.bin because its ramdisk (tmp) is full.

None of the VC features will be applicable if the host is in not responding state.
VM backup will get impacted.
Upgrade via VUM/CLI will fail.

Cause

vm-support will execute nvmelogs.sh to collect nvme telemetry and persistent event logs. nvme logs occupy large space. After the vm-support collection is completed, they will be deleted.

The /tmp is only a temporary folder to store the intermediate files. In the end these files will be saved in the vm-support bundle.

Resolution

This issue is fixed in 7.0 U3q. Please refer to VMware ESXi 7.0 Update 3q Release Notes for more details.

Workaround for the version below VMware ESXi 7.0 Update 3q :

1.From CLI,Execute "vm-support -e Storage:NVMe" command instead of "vm-support" to disable nvmelogs when collecting vm-support on ESXi host.
2.From vcenter GUI,deselect NVMe manifest(There is NVMe under Storage) while generating support bundle from vcenter