Virtual machine in invalid state after power outage on ESXi host
search cancel

Virtual machine in invalid state after power outage on ESXi host

book

Article ID: 443916

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • After a power outage or unexpected host reboot, a virtual machine (VM) on ESXi host or vCenter exhibits the following symptoms:

    • The VM is listed as Invalid in the inventory and VM name appears as a full datastore path (e.g., /vmfs/volumes/####/.../vmname.vmx).
    • Attempts to unregister or re-register the VM through the UI reverts the VM to invalid state.

  •  /var/run/log/hostd.log of ESXi host contains errors such as:

    <YYYY-MM-DD>T<time> verbose hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] Initial Connect message: Failed to open a local pipe to a local virtual machine '/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx'.
    --> Failed to establish transport connection.
    -->
    <YYYY-MM-DD>T<time> info hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] Failed to load virtual machine
    <YYYY-MM-DD>T<time> warning hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] Failed to load VM from vigor during register Fault cause: vim.fault.GenericVmConfigFault
    -->
    <YYYY-MM-DD>T<time> info hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] Marking VirtualMachine invalid
    <YYYY-MM-DD>T<time> info hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)
    <YYYY-MM-DD>T<time> verbose hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] GetVigorCnx: VM not loaded
    <YYYY-MM-DD>T<time> warning hostd[2110143] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#####-####-####-####-#######/VMFolder/VMName.vmx opID=vim-cmd-##-####user=root] Unable to check for locked state: N3Vim5Fault12InvalidState9ExceptionE Fault cause: vim.fault.InvalidState

Environment

VMware vSphere ESXi
VMware vCenter Server

Cause

The VM configuration file (.vmx) is locked by a stale process or an orphaned .lck file from the previous session. This prevents the host daemon (hostd) from loading the configuration.

Resolution

To resolve this issue, you must manually clear the stale process and file locks via the ESXi command line.

  1. Log in to the ESXi as root via SSH and run:    esxcli vm process list | grep "VM_NAME"

    Example output:


  2. Take a note of the world ID of the virtual machine. or example in step 1 the World ID is 2750434. Terminate the VM process using any one of the following commands:

    esxcli vm process kill -t soft -w WorldID
    or
    esxcli vm process kill -t hard -w WorldID
    or
    esxcli vm process kill -t force -w WorldID

    Note: These command provide different levels of force to stop the virtual machine:

    • Soft is the most graceful
    • Hard performs an immediate shutdown
    • Force should be used as a last resort

  3. Remove file locks:
    1. Remove file locks: Navigate to the VM's directory on the datastore: cd /vmfs/volumes/<Datastore_Name>/<VM_Folder>/
    2. Locate and move any .lck files to a temporary location:  mv *.lck /tmp/

  4. Unregister the invalid VM from the UI and then register it again using the .vmx following https://knowledge.broadcom.com/external/article/335224 .
Powered by Wolken Software