Full ramdisk '/var' causes /var/log/EMU/mili/mili2d.log to stop logging
Article ID: 318685
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
- Host not responding
- Connecting to the host ESXi you see /var or /tmp folder at 100%
- The /var/log/EMU/mili/mili2d.log contains these messages:
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:MILI_enumerate_elx_nics:Failed to initialize USer Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:could not open device node /vmfs/devices/char/vmkdriver/be_esx_nic
YYYY:MM:DD hh:mm:ss 5x3x2x7, CRITICAL:backend_init:OneConnect Adapter Not Found.
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:rename all the configuration files!
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:MILI_enumerate_elx_nics:Failed to initialize USer Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:could not open device node /vmfs/devices/char/vmkdriver/be_esx_nic
YYYY:MM:DD hh:mm:ss 5x3x2x4, CRITICAL:backend_init:OneConnect Adapter Not Found.
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:rename all the configuration files!
YYYY:MM:DD hh:mm:ss 5x3x2x1, ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:MILI_enumerate_elx_nics:Failed to initialize USer Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:could not open device node /vmfs/devices/char/vmkdriver/be_esx_nic
YYYY:MM:DD hh:mm:ss 5x3x2x7, CRITICAL:backend_init:OneConnect Adapter Not Found.
YYYY:MM:DD hh:mm:ss 5x3x2x7, ERROR:rename all the configuration files!
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:MILI_enumerate_elx_nics:Failed to initialize USer Init with status = 19
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:could not open device node /vmfs/devices/char/vmkdriver/be_esx_nic
YYYY:MM:DD hh:mm:ss 5x3x2x4, CRITICAL:backend_init:OneConnect Adapter Not Found.
YYYY:MM:DD hh:mm:ss 5x3x2x4, ERROR:rename all the configuration files!
YYYY:MM:DD hh:mm:ss 5x3x2x1, ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19
- In the vmkernel.log, you see:
cpu16:7x4x6xx5)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
cpu14:7x3x4xx6)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/hp-bbUsg.txt because its ramdisk (tmp) is full.
cpu14:7x3x4xx06)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/hp-mem.txt because its ramdisk (tmp) is full.
cpu14:7x3x4xx6)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/hp-bbUsg.txt because its ramdisk (tmp) is full.
cpu14:7x3x4xx06)WARNING: VisorFSRam: 233: Cannot extend visorfs file /tmp/hp-mem.txt because its ramdisk (tmp) is full.
- Connected virtual machine console you see error:
A server error occurred.You have reached the maximum number of connected consoles: 40. Please contact your administrator.
Check the vSphere Web Client server logs for details.
Check the vSphere Web Client server logs for details.
- Migrating virtual machine using vMotion fails.
- You see this error similar to:
General System Error Occurred.
Note: For issues where the same error occurs, but the AMS data is filling up /tmp, see HPE servers running VMware ESXi version 6.X with Agentless Management Service (AMS) version 11.4.x will become not responding. (78595).
Environment
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
Cause
When /scratch/log/ is unavailable Emulex drivers logs will begin to fill up the /var file system logs.
Emulex drivers might write logs to /var/log/EMU/mili/mili2d.log and fill up the 40 MB /var file system logs of RAM drives. A previous fix changed writes of Emulex drivers to the /scratch/log/ folder instead of to the /var/log/ folder to prevent the issue. When the /scratch/log/ folder is temporarily unavailable, the /var/log/EMU/mili/mili2d.log is still periodically used for logging.
Emulex drivers might write logs to /var/log/EMU/mili/mili2d.log and fill up the 40 MB /var file system logs of RAM drives. A previous fix changed writes of Emulex drivers to the /scratch/log/ folder instead of to the /var/log/ folder to prevent the issue. When the /scratch/log/ folder is temporarily unavailable, the /var/log/EMU/mili/mili2d.log is still periodically used for logging.
Resolution
This issue is resolved in VMware ESXi 6.7 P01
Download link: https://support.broadcom.com/group/ecx/productdownloads?subfamily=VMware+vSphere&tab=Solutions
search for the patch: ESXi670-201912001
Download link: https://support.broadcom.com/group/ecx/productdownloads?subfamily=VMware+vSphere&tab=Solutions
search for the patch: ESXi670-201912001
Workaround:
To workaround this issue：
- Remove the mili2d.log from /tmp folder with the command: rm /var/log/EMU/mili/mili2d.log
- Restart management agents on the affected host by running the command: services.sh restart
- Edit the configuration file: /etc/config/EMU/mili/libmili.conf and set 'MILILogLevel 0' to stop mili log ouput.
- Remove the following vibs if not used and reboot host
esxcli software vib remove --vibname elxnet
esxcli software vib remove --vibname elxiscsi
esxcli software vib remove --vibname elx-esx-libelxima.so
esxcli software vib remove --vibname elxiscsi
esxcli software vib remove --vibname elx-esx-libelxima.so
Additional Information
Feedback
Yes
No
Powered by
