ESXi host becomes unresponsive when attempting a vMotion migration or a configuration change
search cancel

ESXi host becomes unresponsive when attempting a vMotion migration or a configuration change

book

Article ID: 328535

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

Symptoms:

  • ESXi host becomes unresponsive after attempting to migrate a virtual machine from VMware vCenter Server.
  • Making a configuration change to the ESXi host renders the host unresponsive.
  • Migration fails at 13%.
  • Some of the virtual machines in the inventory become invalid.
  • vpxa fails to start.
  • Unable to power on a virtual machine.
  • In the vpxa.log file, there are entries similar to:

    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 warning 'Libs'] vmware-vpxa: CnxUnix_UseConfigFile lock of /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8 failed: No space left on device.
    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Cnx_UseConfigFile(vmware-vpxa, vpxa-nfc) failed
    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Error setting up NFC server listener
     
  • When running tasks such as a vMotion migration or powering on a virtual machine, there is this error:

    A general system error occurred: Timed out waiting for vpxa to start
     
  • The vSphere HA agent for the host reports the error:

    The vSphere HA agent is not reachable from vCenter Server.
     
  • Deploying an OVF template fails with the error:

    A general system error occurred. Failed to create journal file provider
     
  • In the /var/log/vmkernel.log file of the host, there are entries similar to:

    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589546_6_3_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/run/vmware/tickets/vmtck-520db3d3-3305-37 for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589576_6_4_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.

    ...

    WARNING: VisorFSRam: 233: Cannot extend visorfs file /var/spool/snmp/1381771804_6_401_6876.trp because its ramdisk (snmptraps) is full.
    WARNING: VisorFSRam: 233: Cannot extend visorfs file /var/spool/snmp/1381772106_6_401_6876.trp because its ramdisk (snmptraps) is full.

    ...

    User: 2742: wantCoreDump : snmpd-enabled : 0

     
  • Similar warnings are observed in the Events tab when connected directly to the ESXi host via the vSphere Client.
  • Opening a virtual machine console from the vSphere Client fails with the error:

    Unable to contact the MKS
     
  • When connected to the ESXi host using SSH and run the stat -f filesystem_name command, it is evident the host is out of inodes.
  • In the Event viewer of vCenter Server, there is the error:

    Cannot install the vCenter agent service. Cannot upload agent.
     
  • Running the stat -f filesystem_name command displays output similar to:

    Inodes: Total: 0 Free: 0

    Note: For more information on connecting to an ESXi host via SSH, see Using ESXi Shell in ESXi (311213).
     
  • Storage vMotion fails and vCenter Server reports an error indicating that the RAM disk is full.
  • In the /var/log/vmkernel.log file, there is an error similar to:

    VisorFSRam: 233: Cannot extend visorfs file /var/log/hpHelper.log because its ramdisk (root) is full.



Cause

This issue occurs when SNMPD is enabled and the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files.

Similar symptoms may also occur on HP ProLiant Gen8 servers running ESXi 5.x, where the /var/log/hpHelper.log file fills the ESXi 5.x RAMDisk.
 
For more information, see:
 
Note: To help identify which files are consuming space, see ESX error: No free space left on device (342658).
 
Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.

Resolution

 

 
To work around this issue:
  1. Connect to the ESXi host using SSH. For more information, see Using ESXi Shell in ESXi (311213).
  2. Check if SNMP is creating too many .trp files in the /var/spool/snmp directory on the ESXi host by running the command:

    ls /var/spool/snmp | wc -l

    Note: If the output indicates that the value is 2000 or more, this may be causing the full inodes.
     
  3. Note: Ensure there are backup copy of the files in a shared datastore before deleting the trap files.
Delete the .trp files in the /var/spool/snmp/ directory by running the commands:

# cd /var/spool/snmp
# for i in $(ls | grep trp); do rm -f $i;done

 

  1. Change directory to /etc/vmware/ and back up the snmp.xml file by running the commands:

    # cd /etc/vmware
    # mv snmp.xml snmp.xml.bkup
     
  2. Create a new file named snmp.xml and open it using a text editor. 
  3. Copy and paste these contents to the file:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <config>
    <snmpSettings><enable>false</enable><port>161</port><syscontact></syscontact><syslocation></syslocation>
    <EnvEventSource>indications</EnvEventSource><communities></communities><loglevel>info</loglevel><authProtocol></authProtocol><privProtocol></privProtocol></snmpSettings>
    </config>
     
  4. Save and close the file.
  5. Reconfigure SNMP on the affected host by running the command:

    # esxcli system snmp set –-enable=true
     
  6. To confirm the SNMP services are running normally again, run the command:

    # esxcli system snmp get

    Here is an example of the output:
    /etc/vmware # esxcli system snmp get
    Authentication: Communities: Enable: true Engineid: 00000063000000a10a0121cf Hwsrc: indications Loglevel: info Notraps: Port: 161 Privacy: Remoteusers: Syscontact: Syslocation: Targets: Users: V3targets:


To ensure that the issue does not recur, temporarily disable snmpd to stop logging. To stop the snmpd service, run this command:

# /etc/init.d/snmpd stop


Additional Information

Note: In the event that the host has run out of inodes, attempt to stop vpxa on the host to free up an inode:

  1. Connect to the host with the vSphere Client.
  2. Click Configuration > Security Profile.
  3. Under Services, click Properties.
  4. Click vpxa > Options.
  5. Under Startup Policy, click Stop.
  6. Click OK.
  7. Click OK.
After vpxa has stopped, an inode is freed up so that it is possible to enable the ESXi Shell and obtain command line access to remove the .trp files as described in the Resolution section. For more information on using the ESXi Shell, see Using ESXi Shell in ESXi (311213).


ESX error: No free space left on device (342658)
Using ESXi Shell in ESXi (311213)
ESXi ramdisk full due to /var/log/hpHelper.log file size (318858)


Powered by Wolken Software