Agentless Management Service (AMS) version 11.4.x causing a full ams-bbUsg.txt and host not responding

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• An ESXi host upgrade fails due to /tmp being full.
• An ESXi host is not responding in the vCenter Server inventory.
• The ams-bbUsg.txt file consumes a large amount of /tmp
• In the vobd.log, you see similar to:

YYYY:MM:DD hh:mm:ss: [VisorfsCorrelator] 9x6x6x2x3x3x8us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
YYYY:MM:DD hh:mm:ss: [VisorfsCorrelator] 9x6x6x2x7x4x1us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
YYYY:MM:DD hh:mm:ss: [VisorfsCorrelator] 9x6x6x2x8x5x5us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/ams-bbUsg.txt because its ramdisk (tmp) is full.
YYYY:MM:DD hh:mm:ss: [VisorfsCorrelator] 9x6x6x1x3x1x4us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/ams-sysmod.txt because its ramdisk (tmp) is full.
YYYY:MM:DD hh:mm:ss: [VisorfsCorrelator] 9x6x8x6x4x0x6us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'tmp' is full.  As a result, the file /tmp/ams-sysmod.txt could not be written.

• In the vmkernel.log, you see similar to:

YYYY:MM:DD hh:mm:ss cpu0:x4x7x8x3)WARNING: VisorFSRam: 206: Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
YYYY:MM:DD hh:mm:ss cpu2:x1x0x8x4)MemSchedAdmit: 470: Admission failure in path: tmp/tmp
YYYY:MM:DD hh:mm:ss cpu2:x1x0x8x3)MemSchedAdmit: 477: tmp (2409) extraMin/extraFromParent: 1/1, tmp (2408) childEmin/eMinLimit: 65536/65536
YYYY:MM:DD hh:mm:ss cpu2:x2x0x6x0)WARNING: VisorFSRam: 206: Cannot extend visorfs file /tmp/ams-bbUsg.txt because its ramdisk (tmp) is full.
YYYY:MM:DD hh:mm:ss cpu20:x2x9x1x3)ScsiDeviceIO: 3068: Cmd(0x45a29fb873c0) 0x1a, CmdSN 0x30e8a1 from world 0 to dev "mpx.vmhba1:C2:T1088:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
YYYY:MM:DD hh:mm:ss cpu6:x4x7x2x3x)MemSchedAdmit: 470: Admission failure in path: tmp/tmp
YYYY:MM:DD hh:mm:ss cpu6:x4x7x2x3x)MemSchedAdmit: 477: tmp (2409) extraMin/extraFromParent: 1/1, tmp (2408) childEmin/eMinLimit: 65536/65536

• vMotion of a virtual machine fails
• You see an error similar to:

General System Error Occurred.

Note: A similar issue can occur if the mili2d.log fills up ramdisk at /var/, see The ramdisk '/var' is full. As a result, the file /var/log/EMU/mili/mili2d.log could not be written. (65193)

Environment

VMware vSphere ESXi 6.7
VMware vSphere 6.5.x

Cause

HPE servers running VMware ESXi version 6.X with Agentless Management Service (AMS) version 11.4.x will enter not responding state because AMS data fills up /tmp.

Resolution

A resolution is available at https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-a00073323en_us

Workaround:
To work around this issue, clear the ams-bbUsg.txt file contents:

1. Connect to affected host through SSH session.
2. List the files in /tmp using du -sh *

[root@VMwareESXi-2:/tmp] du -sh *
255.6M  ams-bbUsg.txt
4.0K    ams-ent.txt
4.0K    ams-fc.txt
4.0K    ams-fsys.txt
64.0K   ams-hrswrun.txt
4.0K    ams-ip.txt
0       ams-ipv4.txt

3. Empty the ams-bbUsg.txt file with this command: echo > ams-bbUsg.txt

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_a38161c3e8674777a8c664e05a#tab-history

Impact/Risks:

• None of the VC features will be applicable if the host is in not responding state.
• VM backup will get impacted.
• Upgrade via VUM/CLI will fail.