Determining why a VM was Powered off / Restarted / Rebooted
search cancel

Determining why a VM was Powered off / Restarted / Rebooted

book

Article ID: 339691

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

  • Clients connected to a virtual machine are no longer accessible
  • Applications dependent upon services running in a virtual machine report errors
  • A virtual machine no longer responds to network connections or to user interaction at the console
  • A virtual machine has been turned off or rebooted unexpectedly
  • The vmware.log for the guest VM may show the error:

PIIX4: PM Soft Off. Good-bye

  • The Windows event viewer for the Windows guest VM may show the error:

User sent the guest OS shutdown

  • The hostd.log for the host of the guest VM may show the error:

Info hostd : State Transition (VM_STATE_ON -> VM_STATE_SHUTTING_DOWN)

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

  1. Verify the location of the virtual machine log files:
    1. Open the vSphere Client and connect to the vCenter Server.
    2. Provide administrative credentials when prompted.
    3. Go to the Hosts & Clusters view.
    4. Select the virtual machine in the Inventory.
    5. Click the Summary tab.
    6. Click Edit Settings.
    7. Click the Options tab.
    8. Select General Options.
    9. Record the Virtual Machine Working Location on the right side.
       
  2. Open the virtual machine log file:
    1. Open a browser and enter the URL http://<vCenter_hostname>.
      Where, <vCenter_hostname> is the IP or fully qualified domain name for the vCenter Server.
    2. Provide administrative credentials when prompted.
    3. Click the Browse datastores in the VirtualCenter inventory link.
    4. Navigate the web pages until the appropriate datacenter and datastore are reached, and folder as noted in step 1.
    5. Click the link to the appropriate log file, and open it with a preferred editor.

      Note: A new vmware-#.log file is created during a hard reset, power on, or vMotion of the virtual machine.
      Use the dates in the Last Modified column to determine the logs, which recorded the outage.
       
  3. Record the name of the VMware ESXi host that was running the virtual machine when it abruptly rebooted or shut down. The second line of the vmware.log file contains an entry that identifies the host. For example:

             [YYYY-MM-DDTHH:MM:SS] vmx| Hostname=esx02.example.com

  4. Determine if the virtual machine was powered off or restarted by the guest operating system:
     
    • A Guest OS initiated reset generates the following event in vmware.log:

      [YYYY-MM-DDTHH:MM:SS]| vcpu-0| CPU reset: soft
      [YYYY-MM-DDTHH:MM:SS]| vcpu-0| Chipset: The guest has requested that the virtual machine be hard reset.
      [YYYY-MM-DDTHH:MM:SS]| vcpu-0| I125: Chipset: Issuing power-off request...

      To continue troubleshooting, contact the operating system vendor to determine why the guest operating system initiated the reset. If the operating system vendor suspects that the outage was caused by VMware Tools or the virtual hardware, contact VMware support. If it is determined that the operating system within the virtual machine reported a blue screen, kernel panic, or equivalent, see Identifying critical Guest OS failures within virtual machines. 

    • A Guest OS initiated halt generates this event in vmware.log:
      [YYYY-MM-DDTHH:MM:SS] vcpu-0| PIIX4: PM Soft Off. Good-bye.

      To continue troubleshooting, contact the operating system vendor to determine why the guest operating system initiated the halt. If the operating system vendor suspects that the outage was caused by VMware Tools or the virtual hardware, contact VMware support. If it is determined that the operating system within the virtual machine reported a blue screen, kernel panic, or equivalent, see Identifying critical Guest OS failures within virtual machines.

  5. Determine if the virtual machine was powered off or restarted by a user or vSphere API call:
     
    • A user or API initiated request to reset a virtual machine generates this event in vmware.log:
      [YYYY-MM-DDTHH:MM:SS] vcpu-0| CPU reset: hard
       
    • A user or API initiated request to power off a virtual machine generates this event in vmware.log:
      [YYYY-MM-DDTHH:MM:SS] vmx| MKS local poweroff
       
    • A user or API initiated request to shutdown a virtual machine generates this event in vmware.log:
      [YYYY-MM-DDTHH:MM:SS] vmx| I120: Tools: sending 'OS_Halt' (state = 1) state change request
      and:
      [YYYY-MM-DDTHH:MM:SS] vcpu-0| I120: PIIX4: PM Soft Off. Good-bye.
       
    • A user or API initiated request to restart the guest operating system in a virtual machine generates this event in vmware.log:
       
      [YYYY-MM-DDTHH:MM:SS] vmx| I120: Tools: sending 'OS_Reboot' (state = 2) state change request
      and:
      [YYYY-MM-DDTHH:MM:SS] vcpu-0| I120: CPU reset: soft (mode 1)
       
  6. Determine if the virtual machine reported a backtrace. A backtrace generates logs similar to this in vmware.log:

    [YYYY-MM-DDTHH:MM:SS]: vmx| Caught signal 6 -- tid 101087
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: eip 0x1019e731 esp 0xbf5ffa9c ebp 0xbf5ffac8
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: eax 0x0 ebx 0x18adf ecx 0x6 edx 0x10054a00 esi 0x18adf edi 0x6
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffa9c : 0x10048dab 0x00018adf 0x00000006 0xbf5ffac8
    [YYYY-MM-DDTHH:MM:SS]: vmx| SIGNAL: stack 0xbf5ffaac : 0x10048d63 0xbf5ffae8 0x1000b8a0 0xbf5ffac8

    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace:
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[0] 0xbf5ff858 eip 0x805ae40
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[11] 0xbf5ffc98 eip 0x81895e7
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[12] 0xbf5ffcb8 eip 0x80cd940
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[13] 0xbf5ffcc8 eip 0x80c4f08
    [YYYY-MM-DDTHH:MM:SS]: vmx| Backtrace[18] 00000000 eip 0x804de11
    [YYYY-MM-DDTHH:MM:SS]: vmx| Unexpected signal: 6.

    To continue troubleshooting the backtrace, see Interpreting virtual machine monitor and executable failures (339468)
     
  7. Determine if the virtual machine operating system had a system fault. The vmware.log will contain entries similar to:
     
    svga| I120: WinBSOD: ( 1) `A problem has been detected and Windows has been shut down to prevent damage '
    svga| I120: WinBSOD: ( 2) `to your computer.'
    svga| I120: WinBSOD: ( 4) `If this is the first time you've seen this Stop error screen,'
    svga| I120: WinBSOD: ( 5) `restart your computer. If this screen appears again, follow'
    svga| I120: WinBSOD: ( 6) `these steps:'
    svga| I120: WinBSOD: ( 8) `Disable or uninstall any anti-virus, disk defragmentation'
    svga| I120: WinBSOD: ( 9) `or backup utilities. Check your hard drive configuration, '
    svga| I120: WinBSOD: (10) `and check for any updated drivers. Run CHKDSK /F to check '
    svga| I120: WinBSOD: (11) `for hard drive corruption, and then restart your computer. '
    svga| I120: WinBSOD: (13) `Technical information: '
    svga| I120: WinBSOD: (15) `*** STOP: 0x00000024 (0x00000000001904FB,0xFFFFF880069B2CE8,0xFFFFF880069B2540,0'
    svga| I120: WinBSOD: (16) `xFFFFF880014F5FA2)

    To continue troubleshooting, contact the operating system vendor to determine why the guest operating system encountered a BSOD. If the operating system vendor suspects that the outage was caused by VMware Tools drivers or the virtual hardware, contact VMware support. If it is determined that the operating system within the virtual machine reported a blue screen, kernel panic, or equivalent, see Identifying critical Guest OS failures within virtual machines. Refer to Converting a snapshot file to memory dump using the vmss2core tool to create a memory dump of the VM to be analyzed by the operating system vendor.
  8. To determine if the virtual machine was restarted by High Availability Virtual Machine Monitoring, review the vmware.log for the affected virtual machine, noting entries similar to:

    vmx| Vix: [104333 vmxCommands.c:457]: VMAutomation_Reset. Trying hard reset

    For more information, see Determining if a High Availability Virtual Machine Monitoring event caused a virtual machine to reboot

Additional Information

Powered by Wolken Software