Troubleshooting single virtual machine failures on an ESX/ESXi host
search cancel

Troubleshooting single virtual machine failures on an ESX/ESXi host

book

Article ID: 309782

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article guide you through the process of troubleshooting a single virtual machine failing on an ESX/ESXi host. This article is primarily suited to troubleshooting repetitive failures that you do not know the cause of. If the failure is reproducible, that is, one that can be repeated by following a sequence of steps, follow the instructions at the end of this knowledge base article to gather the support script data and file a support request. If the failure is a one-off, gather the support script data and file a support request.
Include as much information about your environment and what was happening at the time of the failure, as possible. You must be aware that virtual machine failures may be caused by factors outside of VMware and the cause is not always evident from the support script data.


Symptoms:
  • The guest operating system has terminated unexpectedly
  • The virtual machine is not accessible
  • A blue screen with a Stop error code may be visible on the console
  • An error including the term kernel panic is visible on the console
  • You see one or more of the following errors:
    • BAD_POOL_HEADER
    • KMODE_EXCEPTION_NOT_HANDLED
    • PAGE_FAULT_IN_NONPAGED_AREA
    • STOP: 0x00000050 (0xFFFFFFF8,0x00000000,0xF9CF5C88,0x00000000)
    • STOP: 0x00000019 (0x00000000,0xC00E0FF0,0xFFFFEFD4,0xC0000000)
    • Unknown inaccessible
    • SCSI: 4506: Cannot find a path to device vmhbax:x:x in a good state
    • WARNING: LVM: 4844: vmhbaH:T:L:P detected as a snapshot device. Disallowing access to the LUN since resignaturing is turned off.
    • Date esx vmkernel: Time cpu3: 10340 SCSI: 5637: status SCSI LUN is in snapshot state, rstatus 0xc0de00 for vmhbax:x:x. residual R 999, CR 8-, ER3
    • Date esx vmkernel: Time cpu3: world ID SCSI 6624: Device vmhbax:x:x. is a deactivated snapshot


Environment

VMware vSphere ESXi 5.1
VMware ESX 4.1.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Embedded
VMware ESX Server 3.0.x
VMware ESX Server 3.5.x
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX Server 2.5.x
VMware vSphere ESXi 5.5
VMware ESX 4.0.x

Resolution

Validate that each troubleshooting step below is true for your environment. Each step will provide instructions or a link to a document, in order to eliminate possible causes and take corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Do not skip a step.
  1. Verify that the virtual machine is not in an unresponsive state.

    During an unresponsive state, the operating system seems to be paralyzed, no error messages are displayed, and the screen freezes or the application does not respond to users' actions. Keyboard input or mouse clicking has no effect, regardless of where the cursor is placed, but the operating system is still running. Unlike a failure, sometimes an unresponsive system resolves itself, and the application resumes its normal execution without your involvement.

    A failure is a situation where the operating system has terminated and is no longer running. There may be a diagnostic screen or error message visible in its place.

    Note: There is a difference between a virtual machine failing and the guest operating system failing. If the virtual machine fails, it powers off and vmware-core files may have been created in the virtual machines host directory. Checking the vmware.log file you may see an entry similar to:

    Sep 13 19:58:46: vcpu-1| MONITOR PANIC: ASSERT failed
    Sep 13 19:58:46: vcpu-1| Core dump with build build-10104
    Sep 13 19:58:46: vcpu-1| Writing monitor corefile
    "/root/vmware/vm1/vmware-core0.gz"|

    Be aware of this difference when opening a support request with VMware support.

  2. Verify that the guest operating system you are experiencing problems with, is fully certified.

    The     Guest Operating System Installation Guide outlines which guest operating systems are certified for each version of ESX/ESXi.

    If your guest operating system is not listed, the following steps may help to resolve the issue, but be aware that problems may be encountered in an uncertified guest operating system.

  3. Verify that you can access the storage hosting the virtual machine.

    Your virtual machine may fail if the LUN on which it is stored becomes unavailable.

    To check this:
    1. Open a command prompt on your ESX host. For more information, see Opening a command or shell prompt (1003892).
    2. Type vmware-cmd -l to list the location of the configuration files for the virtual machines registered on that ESX/ESXi host.
    3. Find the virtual machine that has just failed and copy the location.

      For example, /vmfs/volumes/46b2f3eb-ced4c7d8-c1d2-111122223333/vm1/vm1.vmx
    4. Type ls and paste the location you just copied removing the <filename.vmx> portion.

      For example, ls /vmfs/volumes/46b2f3eb-ced4c7d8-c1d2-111122223333/vm1/
    5. Press Enter.
    6. If the files associated with your virtual machine (VMDK, VMX, NVRAM) are listed, you are able to access the storage hosting your virtual machine.

      If not, see       Identifying Fibre Channel, iSCSI, and NFS storage issues on ESX/ESXi hosts (1003659).

  4. Verify no software changes have been made that may have caused the failure. For more information, see Identifying critical Guest OS failures within virtual machines (1003999).

  5. Verify no hardware changes have been made that may have caused the failure. If you have recently made any changes to your virtual machines hardware configuration, back them out temporarily for testing purposes. For more information, see Verifying the Virtual Hardware configuration of a virtual machine (1003986) .
Note: If your problem still exists after trying the steps in this article, please:


Additional Information

Identifying Fibre Channel, iSCSI, and NFS storage issues on ESX/ESXi hosts
Verifying the Virtual Hardware configuration of a virtual machine
Identifying critical Guest OS failures within virtual machines

Powered by Wolken Software