Troubleshooting a virtual machine that has stopped responding (VM hang/freeze)
search cancel

Troubleshooting a virtual machine that has stopped responding (VM hang/freeze)

book

Article ID: 326252

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

An unresponsive virtual machine does not respond to any connection attempts and may be unable to respond to any attempts to power cycle it. There are a variety of reasons a virtual machine can end up in an unresponsive state.
This article is meant as a helps for identifying and resolving common causes and, when resolved, returning the virtual machine to an operational state.

While in most cases it will be possible to remediate such issues and bring the VM back in a working state by powering it off and on again as outlined in Powering off an unresponsive virtual machine on an ESXi host, this will prevent collection and analysis of information which could assist with determining the root cause of the outage.

This article assumes that the issue is currently occurring. If troubleshooting an issue that occurred in the past, some required information may be unavailable.

Virtual machines can become unresponsive/freeze/hang in the same way as physical systems, and might show a variety of symptoms such as:

  • Virtual machines are unresponsive to tasks or appear to be frozen/hang.
  • Tasks performed on the virtual machine fail, timeout, or do not start.
  • Virtual machine does not produce network or disk traffic.
  • Virtual machine does not allow access via RDP, vCenter Server virtual machine console screen, or other connection methods.
  • Virtual machines are unreachable over the network.
  • The VSCSI I/O warning counters in the esxtop indicates inactivity.
  • Virtual machines may report an invalid state
  • The screen is frozen, and no actions are possible.
  • The affected virtual machines have a black console with an error like:
    Virtual machine screen is black and does not refresh.
  • One or more of the following errors might be seen:
    /init: /init: 151: Syntax error: 0xforce=panic
    Kernel panic - not syncing: Attempted to kill init!
    PAGE_FAULT_IN_NONPAGED_AREA
    (or any other kind of Windows Bluescreen/BSOD)

Environment

  • VMware vSphere ESXi 7.0.x
  • VMware vSphere ESXi 8.0.x
  • VMware ESX 9.x

Resolution

The services a virtual machine (VM) provides may become unresponsive or unreachable due to several causes. These include problems with the applications or guest OS, issues with the Virtual Machine Monitor (VMM) or virtual devices, resource contention on the host, or underlying storage and networking infrastructure issues.
If the guest OS is producing any activity, it has to be assumed running successfully. In this case, unresponsiveness is likely due to a connectivity problem, resource contention, or a higher-level component (such as an application or service) running within the guest OS.

Phase 1: Validate the scope

Is is important to document accurate symptoms and to understand the scope of the problem. Work through the following checks:

  1. Confirm VM unresponsiveness: A VM may stop responding via one interface but function correctly on others. See: Confirming whether a virtual machine is unresponsive.
    Note: If the VM is responsive but performing poorly, refer to Troubleshooting ESX/ESXi virtual machine performance issues.
  2. Verify the VM power state: Ensure the VM is actually powered on. If it powered off unexpectedly, power it back on and investigate the shutdown cause. Refer to Powering on virtual machine from ESXi host (CLI) and Determining Why a Virtual Machine was Unexpectedly Powered Off, Restarted, or Rebooted.
    Note: If the VM cannot be powered on, refer to Troubleshooting a Virtual Machine that is unable to Power ON.
  3. Determine the number of affected VMs: Are multiple VMs affected or just one? If multiple, look for shared infrastructure dependencies (e.g., specific datastores or hosts).
  4. Test console interaction: Check if the guest OS responds to interaction at the VM console. If it does, the issue is likely isolated to the guest OS or internal applications (see Troubleshooting virtual machine network connection issues).
  5. Check for critical errors: Determine if the guest OS has reported critical errors to the console and is in a halted state (see Identifying critical Guest OS failures within virtual machines).
  6. Check host responsiveness: Determine if the ESX/ESXi host is also unresponsive. If it is, the scope is larger than a single VM. Refer to Determining why an ESXi host does not respond to user interaction at the console.

Phase 2: Identify the Cause

Once it was established that the VM is unresponsive at both the virtual console and via the network all the while the host remains responsive, investigate the underlying cause:

  1. Review recent operations: Did a specific task trigger the issue? For example, snapshot and vMotion operations briefly "stun" a VM while memory state is copied to disk or across the network (see Virtual machine becomes unresponsive or inactive when taking memory snapshot).
  2. Verify configuration: Review VM and host configurations for common errors that cause unresponsiveness, such as waiting for an unavailable resource.
  3. Validate backing infrastructure: VMs depend on functional storage and networking. If the backing infrastructure fails, the virtual hardware presented to the guest OS is impacted (see ESX Server virtual machines stop responding due to shared storage connectivity issues, Verifying that ESX/ESXi virtual machine storage is accessible and Troubleshooting virtual machine network connection issues).
  4. Check resource availability: A problem with CPU/Memory availability or scheduling can cause unresponsiveness. Check if the VM is blocking on unavailable resources or spinning at 100% vCPU utilization. Refer to Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison.

Phase 3: Action Plan

At this stage it has been verified that the host is responsive, there are no storage- or network outages, and that the guest operating system has not failed with a critical error. Yet the virtual machine remains unresponsive.
Choose the appropriate action plan below based on the suspected architectural layer:

  1. The issue is isolated to the Guest OS (or %RUN is high)

    If the VMM is functioning correctly, the guest OS may be hanging just as it would on physical hardware. See Troubleshooting vm not responding issues.
    1. Collect performance data while the problem is actively occurring / present.
    2. Attempt to manually induce a kernel panic inside the guest OS, in order to collect internal state information:
    3. Suspend the virtual machine to collect its internal state. For more information, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information.
      • Hardware version 10 or older: Collect the .vmss file.
      • Hardware version 11 or newer: Collect both the .vmss and .vmem files.>
        Note: If a stuck or failed management task prevents suspension, see Restarting Management Agents in ESXi.
    4. Collect diagnostic logs from the ESX/ESXi host running the VM. For more information see Collecting diagnostic information for VMware products.
    5. Power-cycle the virtual machine (try Power off → Power ON first, then Reset).
    6. Contact Broadcom support and provide the performance data, suspend state files and host logs collected in steps 1,3 and 4. For more information, see also Creating and managing Broadcom cases.

  2. The issue is isolated to the Virtual Machine Monitor (or %VMWAIT is high)

    If attempts to suspend the VM fail, but the host is stable, force a crash.
    1. Collect performance data while the problem is activaly occurring/present.
    2. Forcefully crash the virtual machine to collect information about its internal state.
    3. Contact Broadcom support and provide the performance data and VM crash state information collected in steps 1 and 2. For more information, see also Creating and managing Broadcom cases.

  3. The issue is isolated to the VMkernel (VMM issues, but suspend/crash fails)

    If the virtual machine cannot neither be suspended nor crashed, this indicates a problem with the VMMonitor component of the ESXi/ESX host. An NMI (active crash of the host) will be required in order to gather information required to isolate the cause.
    1. Collect performance data while the problem is activaly occurring/present.
    2. Evacuate unaffected VMs to other hosts using vMotion. Change the DRS mode in the cluster to manual or half-automated to prevent DRS from moving VMs back.
    3. Configure the host to panic upon receiving a non-maskable interrupt (NMI), then issue an NMI using vendor-specific serial management interface. See Troubleshooting unresponsive ESXi hosts using hardware NMI facilities for more information.
    4. Capture the Purple Diagnostic Screen (PSOD) by taking a screenshot or photograph of the console once the diagnostic dump completes, then restart the host.
    5. Collect diagnostic logs from the ESX/ESXi host running the VM. For more information see Collecting diagnostic information for VMware products.
    6. Contact Broadcom support and provide the performance data, PSOD screenshot, and host logs collected in steps 1, 4, and 5. For more information, see also Creating and managing Broadcom cases.

  4. It is unclear which of the above scenarios might be present

    If the issue cannot be isolated to any of the above scenarios, see What to collect when a virtual machine is hung or frozen on an ESXi host.
Powered by Wolken Software