Powering off VM fails with error "The operation is not allowed in the current state. The operation cannot be performed because VM migration is in progress".

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Unable to Power-off the VM.
• Powering off VM fails with error: "The operation is not allowed in the current state. The operation cannot be performed because VM migration is in progress".
• The vCenter server do not show this pending migration task. 

Environment

VMware vSphere ESXi 8.x

Cause

The VM power off task fails due to an existing stale migration task.

Cause Validation:

• The var/log/vmware/vpxd/vpxd.log file on the vCenter Server confirms the guest shutdown failure due to a pending migration task
  YYYY-MM-DDTHH:MM.SSSZ error vpxd[469014] [Originator@6876 sub=Default opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6] [VpxLRO] -- ERROR task-74265 -- ########-####-####-####-############(########-####-####-####-############) -- vm-####-- vim.VirtualMachine.shutdownGuest: :vim.fault.InvalidState
--> Result:
--> (vim.fault.InvalidState) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vm.error.vmMigrationInProgress",
-->          arg = <unset>,
-->          message = "The operation can not be performed because VM migration is in progress."
-->       }
-->    ]
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f4aa80f5098, h:19, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-####/vpxa>
]: shutdownGuest
--> Received SOAP response fault from [<<io_obj p:0x000000cbc31151a0, h:21, <TCP '127.0.0.1 : 37285'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: shutdownGuest
--> The operation is not allowed in the current state."
--> }
--> Args:
-->
• The var/run/log/hostd.log file on the host also confirms guest shut down failure due to a pending VM migration task.
  YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102445]: [Originator@6876 sub=Vimsvc.TaskManager opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6-23-0954 sid=529bc107 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask-9-vim.VirtualMachine.shutdownGuest-#######
YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2102446]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/VM_NAME.vmx opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6-23-0954 sid=529bc107 user=vpxuser:VSPHERE.LOCAL\Administrator] Shutting down guest
YYYY-MM-DDTHH:MM.SSSZ Er(163) Hostd[2102446]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/VM_NAME.vmx opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6-23-0954 sid=529bc107 user=vpxuser:VSPHERE.LOCAL\Administrator] Invalid transition requested (VM_STATE_EMIGRATING -> VM_STATE_SHUTTING_DOWN): Invalid state
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102446]: [Originator@6876 sub=Vimsvc.TaskManager opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6-23-0954 sid=529bc107 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask-9-vim.VirtualMachine.shutdownGuest-####### Status error
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102446]: [Originator@6876 sub=Solo.Vmomi opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6-23-0954 sid=529bc107 user=vpxuser:VSPHERE.LOCAL\Administrator] Throw vim.fault.InvalidState
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102446]: [Originator@6876 sub=Solo.Vmomi opID=mkfb95av-81350-auto-1qrs-h5:70032925-d6-23-0954 sid=529bc107 user=vpxuser:VSPHERE.LOCAL\Administrator] Result:
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: --> (vim.fault.InvalidState) {
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: -->          key = "com.vmware.vim.vm.error.vmMigrationInProgress",
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: -->       }
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: -->    ],
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: -->    msg = ""
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2102399]: --> }
• Host cli, confirms no pending tasks.
  • Login to SSH session for the host.
  • Run the below mentioned command and note down the VMID. 
    vim-cmd vmsvc/getallvms 
  • Running the command "vim-cmd vmsvc/get.tasklist VMID", confirms no tasks are pending
    vim-cmd vmsvc/get.tasklist VMID
(ManagedObjectReference) []

Resolution

• To kill the stale vMotion task, restart hostd services on host. 
  
  • Login to SSH session for ESXi host. Run the below command.
    /etc/init.d/hostd restart
• VM now enters into powered-off state. 

If powering on the VM fails, follow the below mentioned procedure:

• • Navigate to the datastore and ensure the VMX file exist and not corrupted. 
  • Login to SSH session for the host.
  • Running the command "vim-cmd vmsvc/getallvms" confirms VM is in invalid state.
    vim-cmd vmsvc/getallvms
Skipping invalid VM '##'
  • Ensure the VMID is same as the VMID collected before powering-off the VM.
  • Run the below mentioned command to ensure no lock on the VMX file
    vmfsfilelockinfo -p /vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME.vmx
vmfsfilelockinfo Version 2.0
Looking for lock owners on "VM_NAME.vmx"
"VM_NAME.vmx" is not locked by any ESX host and is Free
Total time taken : 0.022576133022084832 seconds.
  • Running the command "esxcli vm process list" confirms a stale VM process with World ID ####### still running.
    esxcli vm process list
VM_NAME
   World ID: #######
   Process ID: 0
   VMX Cartel ID: #######
   UUID: ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
   Display Name: ######
   Config File: /vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME.vmx
  • Kill this stale process:
    esxcli vm process kill --type=force --world-id=<WORLD_ID>
  • Restart the hostd and vpxa services.
    
    • /etc/init.d/hostd restart
hostd stopped.
hostd started.
    • /etc/init.d/vpxa restart
vpxa stopped.
vpxa started.
  • Unregister the VM
    vim-cmd vmsvc/unregister <VMID>
  • Re-register the VM
    vim-cmd solo/registervm /vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME.vmx