Determining why a virtual machine was powered off or restarted
search cancel

Determining why a virtual machine was powered off or restarted

book

Article ID: 339691

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides troubleshooting steps to determine why a virtual machine was powered off or rebooted. 

Symptoms:
  • Clients connected to services running in a virtual machine are no longer accessible
  • Applications dependent on services running in a virtual machine are reporting errors
  • A virtual machine no longer responds to network connections
  • A virtual machine no longer responds to user interaction at the console
  • A virtual machine has been turned off unexpectedly
  • A virtual machine has been rebooted unexpectedly
  • In the vmware.log, an error similar to this appears:
    PIIX4: PM Soft Off. Good-bye
  • In the Windows Event Viewer, this message is seen:
    User sent the guest OS shutdown
  • Hostd logs (hostd.log) shows below snippet:
    Info hostd : State Transistion (VM_STATE_ON -> VM_STATE_SHUTTING_DOWN)


Environment

VMware vCenter Server 6.x, 7.x, 8.x
VMware vSphere ESXi  6.x, 7.x, 8.x
 

Resolution

To determine why the virtual machine was powered off or rebooted:
  1. Verify the location of the virtual machine log files:
    1. Open the vSphere Client and connect to the vCenter Server.
    2. Provide administrative credentials when prompted.
    3. Go to the Hosts & Clusters view.
    4. Select the virtual machine in the Inventory.
    5. Click the Summary tab.
    6. Click Edit Settings.
    7. Click the Options tab.
    8. Select General Options.
    9. Record the Virtual Machine Working Location on the right side.
       
  2. Open the virtual machine log file:
    1. Open a browser and enter the URL http://<vCenter hostname>.
      Where, <vCenter hostname> is the IP or fully qualified domain name for the vCenter Server.
    2. Provide administrative credentials when prompted.
    3. Click the Browse datastores in the VirtualCenter inventory link.
    4. Navigate the webpages until the appropriate datacenter and datastore are reached, and folder as noted in step 1.
    5. Click the link to the appropriate log file, and open it with a preferred editor.

      Note: A new vmware-#.log file is created during a hard reset, power on, or vMotion of the virtual machine. Use the dates in the Last Modified column to determine the logs, which recorded the outage.
       
  3. Record the name of the VMware ESXi host that was running the virtual machine when it abruptly rebooted or shut down. The second line of the vmware.log file contains an entry that identifies the host. For example:

             [YYYY-MM-DDTHH:MM:SS] vmx| Hostname=esx02.example.com

  4. Determine if the virtual machine was powered off or restarted by the guest operating system:
     
    • A Guest OS initiated reset generates this event in the virtual machine logs, i.e vmware.log

      [YYYY-MM-DDTHH:MM:SS]| vcpu-0| CPU reset: soft

      [YYYY-MM-DDTHH:MM:SS]| vcpu-0| Chipset: The guest has requested that the virtual machine be hard reset.

      [YYYY-MM-DDTHH:MM:SS]| vcpu-0| I125: Chipset: Issuing power-off request...

      To continue troubleshooting, contact the operating system vendor to determine why the guest operating system initiated the reset. If the operating system vendor suspects that the outage was caused by VMware Tools or the virtual hardware, contact VMware support. If it is determined that the operating system within the virtual machine reported a blue screen, kernel panic, or equivalent, see Identifying critical Guest OS failures within virtual machines (315245). 
    • A Guest OS initiated halt generates this event within the virtual machine logs(vmware.log)

      [YYYY-MM-DDTHH:MM:SS] vcpu-0| PIIX4: PM Soft Off. Good-bye.

      To continue troubleshooting, contact the operating system vendor to determine why the guest operating system initiated the halt. If the operating system vendor suspects that the outage was caused by VMware Tools or the virtual hardware, contact VMware support. If it is determined that the operating system within the virtual machine reported a blue screen, kernel panic, or equivalent, see Identifying critical Guest OS failures within virtual machines (314365).

  5. Determine if the virtual machine was powered off or restarted by a user or vSphere API call:
     
    • A user or API initiated request to reset a virtual machine generates this event in the virtual machine log files:(vmware.log)

      [YYYY-MM-DDTHH:MM:SS] vcpu-0| CPU reset: hard
       
    • A user or API initiated request to power off a virtual machine generates this event in the virtual machine log files:(vmware.log)

      [YYYY-MM-DDTHH:MM:SS] vmx| MKS local poweroff
       
    • A user or API initiated request to shutdown a virtual machine generates this event in the virtual machine log files:(vmware.log)
       
      [YYYY-MM-DDTHH:MM:SS] vmx| I120: Tools: sending 'OS_Halt' (state = 1) state change request
       
      and:
       
      [YYYY-MM-DDTHH:MM:SS] vcpu-0| I120: PIIX4: PM Soft Off. Good-bye.
       
    • A user or API initiated request to restart the guest operating system in a virtual machine generates this event in the virtual machine log files:(vmware.log)
       
      [YYYY-MM-DDTHH:MM:SS] vmx| I120: Tools: sending 'OS_Reboot' (state = 2) state change request
       
      and:
       
      [YYYY-MM-DDTHH:MM:SS] vcpu-0| I120: CPU reset: soft (mode 1)
       
  6. Determine if the virtual machine reported a backtrace. A backtrace generates events similar to this in the virtual machine log file:(vmware.log)

    [YYYY-MM-DDTHH:MM:SS]: vmx| Caught signal 6 -- tid 101087
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: eip 0x1019e731 esp 0xbf5ffa9c ebp 0xbf5ffac8
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: eax 0x0 ebx 0x18adf ecx 0x6 edx 0x10054a00 esi 0x18adf edi 0x6
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffa9c : 0x10048dab 0x00018adf 0x00000006 0xbf5ffac8
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffaac : 0x10048d63 0xbf5ffae8 0x1000b8a0 0xbf5ffac8
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffabc : 0x10054a00 0x00000006 0x0861e1a0 0xbf5ffae8
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffacc : 0x100492c7 0x00000400 0x00000006 0x00000001
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffadc : 0x100492aa 0x1029e5a8 0x00000001 0xbf5ffc18
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffaec : 0x1019fc82 0x00000006 0xbf5ffb00 0x00000000
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffafc : 0x1019fb30 0x00000020 0x00000000 0x00000000
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffb0c : 0x00000000 0x00000000 0x00000000 0x00000000
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace:
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[0] 0xbf5ff858 eip 0x805ae40
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[1] 0xbf5ff928 eip 0x80f735a
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[2] 0xbf5ff9a8 eip 0x80f70e8
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[3] 0xbf5ffa28 eip 0x10048e5b
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[4] 0xbf5ffa94 eip 0x7d0004f
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[5] 0xbf5ffac8 eip 0x1019e731
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[6] 0xbf5ffae8 eip 0x100492c7
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[7] 0xbf5ffc18 eip 0x1019fc82
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[8] 0xbf5ffc38 eip 0x101eed95
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[9] 0xbf5ffc68 eip 0x101eca2d
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[10] 0xbf5ffc78 eip 0x818e87c
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[11] 0xbf5ffc98 eip 0x81895e7
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[12] 0xbf5ffcb8 eip 0x80cd940
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[13] 0xbf5ffcc8 eip 0x80c4f08
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[14] 0xbf5ffcf8 eip 0x80c4b15
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[15] 0xbf5ffd48 eip 0x804fd18
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[16] 0xbf5ffd68 eip 0x804ec14
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[17] 0xbf5ffda8 eip 0x1018c853
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[18] 00000000 eip 0x804de11
    [YYYY-MM-DDTHH:MM:SS]: vmx| Unexpected signal: 6.

    To continue troubleshooting the backtrace, see Interpreting virtual machine monitor and executable failures (339468).
     
  7. Determine if the virtual machine operating system had a system fault. The vmware.log will contain entries similar to:
     
    svga| I120: WinBSOD: ( 1) `A problem has been detected and Windows has been shut down to prevent damage '
    svga| I120: WinBSOD: ( 2) `to your computer.'
    svga| I120: WinBSOD: ( 4) `If this is the first time you've seen this Stop error screen,'
    svga| I120: WinBSOD: ( 5) `restart your computer. If this screen appears again, follow'
    svga| I120: WinBSOD: ( 6) `these steps:'
    svga| I120: WinBSOD: ( 8) `Disable or uninstall any anti-virus, disk defragmentation'
    svga| I120: WinBSOD: ( 9) `or backup utilities. Check your hard drive configuration, '
    svga| I120: WinBSOD: (10) `and check for any updated drivers. Run CHKDSK /F to check '
    svga| I120: WinBSOD: (11) `for hard drive corruption, and then restart your computer. '
    svga| I120: WinBSOD: (13) `Technical information: '
    svga| I120: WinBSOD: (15) `*** STOP: 0x00000024 (0x00000000001904FB,0xFFFFF880069B2CE8,0xFFFFF880069B2540,0'
    svga| I120: WinBSOD: (16) `xFFFFF880014F5FA2)

    To continue troubleshooting, contact the operating system vendor to determine why the guest operating system encountered a BSOD. If the operating system vendor suspects that the outage was caused by VMware Tools drivers or the virtual hardware, contact VMware support. If it is determined that the operating system within the virtual machine reported a blue screen, kernel panic, or equivalent, see Identifying critical Guest OS failures within virtual machines (315245). Refer to Converting a snapshot file to memory dump using the vmss2core tool to create a memory dump of the VM to be analyzed by the operating system vendor.

  8. Ensure that there was no outage due to the unavailability of shared resources.

  9. Determine if the virtual machine was restarted by High Availability Virtual Machine Monitoring.

    Review the vmware.log for the affected virtual machine (/vmfs/volumes/<datastore>/<VM directory>/vmware.log), noting entries similar to:

    vmx| Vix: [104333 vmxCommands.c:457]: VMAutomation_Reset. Trying hard reset

    For more information, see Determining if a High Availability Virtual Machine Monitoring event caused a virtual machine to reboot (337335).



Additional Information

Powered by Wolken Software