ESXi host becomes unresponsive when attempting a vMotion migration or a configuration change
search cancel

ESXi host becomes unresponsive when attempting a vMotion migration or a configuration change

book

Article ID: 328535

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
  • ESXi host becomes unresponsive after attempting to migrate a virtual machine from VMware vCenter Server.
  • Making a configuration change to the ESXi host renders the host unresponsive.
  • Migration fails at 13%.
  • Some of the virtual machines in the inventory become invalid.
  • vpxa fails to start.
  • You are unable to power on a virtual machine.
  • In the vpxa.log file, you see entries similar to:

    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 warning 'Libs'] vmware-vpxa: CnxUnix_UseConfigFile lock of /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8 failed: No space left on device.
    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Cnx_UseConfigFile(vmware-vpxa, vpxa-nfc) failed
    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Error setting up NFC server listener
     
  • When you complete tasks such as a vMotion migration or powering on a virtual machine, you see the error:

    A general system error occurred: Timed out waiting for vpxa to start
     
  • The vSphere HA agent for the host reports the error:

    The vSphere HA agent is not reachable from vCenter Server.
     
  • Deploying an OVF template fails with the error:

    A general system error occurred. Failed to create journal file provider
     
  • In the /var/log/vmkernel.log file of the host, you see entries similar to:

    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589546_6_3_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/run/vmware/tickets/vmtck-520db3d3-3305-37 for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589576_6_4_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.

    ...

    WARNING: VisorFSRam: 233: Cannot extend visorfs file /var/spool/snmp/1381771804_6_401_6876.trp because its ramdisk (snmptraps) is full.
    WARNING: VisorFSRam: 233: Cannot extend visorfs file /var/spool/snmp/1381772106_6_401_6876.trp because its ramdisk (snmptraps) is full.

    ...

    User: 2742: wantCoreDump : snmpd-enabled : 0

     
  • You observe similar warnings in the Events tab when connected directly to the ESXi host via the vSphere Client.
  • Opening a virtual machine console from the vSphere Client fails with the error:

    Unable to contact the MKS
     
  • When you connect to the ESXi host using SSH and run the stat -f filesystem_name command, you see that the host is out of inodes.
  • In the Event viewer of vCenter Server, you see the error:

    Cannot install the vCenter agent service. Cannot upload agent.
     
  • Running the stat -f filesystem_name command displays output similar to:

    Inodes: Total: 0 Free: 0

    Note: For more information on connecting to an ESXi host via SSH, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
     
  • Storage vMotion fails and vCenter Server reports an error indicating that the RAM disk is full.
  • In the /var/log/vmkernel.log file, you see an error similar to:

    VisorFSRam: 233: Cannot extend visorfs file /var/log/hpHelper.log because its ramdisk (root) is full.


Cause

This issue occurs when SNMPD is enabled and the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files.

Similar symptoms may also occur on HP ProLiant Gen8 servers running ESXi 5.x, where the /var/log/hpHelper.log file fills the ESXi 5.x RAMDisk.
 
For more information, see:
 
Note: To help identify which files are consuming space, see ESX error: No free space left on device (1007638).
 
Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.

Resolution

This issue is resolved in ESXi 5.1 Patch 04. For more information, see VMware ESXi 5.1, Patch ESXi510-201404401-BG: Updates esx-base (2070667).
 
To work around this issue:
  1. Connect to the ESXi host using SSH. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
  2. Check if SNMP is creating too many .trp files in the /var/spool/snmp directory on the ESXi host by running the command:

    ls /var/spool/snmp | wc -l

    Note: If the output indicates that the value is 2000 or more, this may be causing the full inodes.
     
  3. Note: Ensure you keep a backup copy of the files in a shared datastore before you delete the trap files.
Delete the .trp files in the /var/spool/snmp/ directory by running the commands:

# cd /var/spool/snmp
# for i in $(ls | grep trp); do rm -f $i;done
 
  1. Change directory to /etc/vmware/ and back up the snmp.xml file by running the commands:

    # cd /etc/vmware
    # mv snmp.xml snmp.xml.bkup
     
  2. Create a new file named snmp.xml and open it using a text editor. For more information, see Editing files on an ESX host using vi or nano (1020302).
  3. Copy and paste these contents to the file:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <config>
    <snmpSettings><enable>false</enable><port>161</port><syscontact></syscontact><syslocation></syslocation>
    <EnvEventSource>indications</EnvEventSource><communities></communities><loglevel>info</loglevel><authProtocol></authProtocol><privProtocol></privProtocol></snmpSettings>
    </config>
     
  4. Save and close the file.
  5. Reconfigure SNMP on the affected host by running the command:

    # esxcli system snmp set –-enable=true
     
  6. To confirm the SNMP services are running normally again, run the command:

    # esxcli system snmp get

    Here is an example of the output:
    /etc/vmware # esxcli system snmp get
    Authentication: Communities: Enable: true Engineid: 00000063000000a10a0121cf Hwsrc: indications Loglevel: info Notraps: Port: 161 Privacy: Remoteusers: Syscontact: Syslocation: Targets: Users: V3targets:

To ensure that the issue does not recur, you can temporarily disable snmpd to stop logging. To stop the snmpd service, run this command:

# /etc/init.d/snmpd stop


Additional Information

Note: In the event that the host has run out of inodes, attempt to stop vpxa on the host to free up an inode:
  1. Connect to the host with the vSphere Client.
  2. Click Configuration > Security Profile.
  3. Under Services, click Properties.
  4. Click vpxa > Options.
  5. Under Startup Policy, click Stop.
  6. Click OK.
  7. Click OK.
After vpxa has stopped, an inode is freed up so that you can enable the ESXi Shell and obtain command line access to remove the .trp files as described in the Resolution section. For more information on using the ESXi Shell, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).

ESXi/ESX error: No free space left on device
Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x
Editing files on an ESX host using vi or nano
Using ESXi Shell in ESXi 5.x and 6.x
ESXi ramdisk full due to /var/log/hpHelper.log file size
VMware ESXi 5.1, Patch ESXi510-201404401-BG: Updates esx-base
vMotion を使用して移行または構成変更しようとすると VMware ESXi 5.x ホストが応答しなくなる
在尝试 vMotion 迁移或配置更改时，VMware ESXi 5.x 主机变得无响应

Powered by Wolken Software