Snapshot Consolidation fails with "Operation not allowed in current state"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Attempting to consolidate a snapshot of a VM fails
• Errors similar to this are presented in the vCenter UI

Cannot create snapshot
Operation not allowed in current state

• The hostd log on the host contains errors similar to the following:

/var/run/log/hostd.log:

YYYY-MM-DDThh:mm:ss.111Z error hostd[2100433] [Originator@#### sub=Vmsvc.vm:/vmfs/volumes/<datastore_id>/<vm_name>/vm_name.vmx opID=esxui-####-#### user=root] Invalid transition requested (VM_STATE_EXPORTING -> VM_STATE_CONSOLIDATE_ALL_DISKS): Invalid state

Environment

vSphere 7.x, 8.x

Cause

This error occurs if the VM is currently undergoing a change in sate as there are related tasks running or queued up for that vm.

Resolution

To resolve this issue, ideally the task(s) need to be allowed some time to continue to proceed with success or time out automatically.

In case the task needs to be cancelled, kindly proceed with caution being thoughtful of the environment and the impact on the same. There are certain tasks like storage vmotion or snapshot deletion/ consolidation tasks which are not recommended to be cancelled due to risk of data loss. If you are unsure, please raise a ticket with Broadcom Support

To cancel the task(s) following steps need to be attempted:

• Cancel the action/task currently running against from the vCenter or Esxi Host ui. 
• If the task cannot be cancelled from the ui, cli method can be attempted:
  • SSH to the host and fetch the vm id:

vim-cmd vmsvc/getallvms |grep <vm_name>

• • Fetch the list of all tasks running on the vm:

vim-cmd vmsvc/get.tasklist <vm_id>

Example output:

(ManagedObjectReference) [
'vim.Task:haTask-<vm_id>-vim.VirtualMachine.<task_name>-####'
]

• • Review the task information: 

vim-cmd vimsvc/task_info “<task_string>”

Example:

vim-cmd vimsvc/task_info haTask-<vm_id>-vim.VirtualMachine.<task_name>-####

Example Output:

(vim.TaskInfo) {
dynamicType = ,
key = "haTask-<vm_id>-vim.VirtualMachine.<task_name>-####",
task = 'vim.Task:haTask-<vm_id>-vim.VirtualMachine.<task_name>-####',
description = (vmodl.LocalizableMessage) null,
name = "vim.VirtualMachine.<task_name>",
descriptionId = "VirtualMachine.<task_name>", 
entity = 'vim.VirtualMachine:<vm_id>', 
entityName = "<vm_name>",  
state = "running",
cancelled = false,
cancelable = false,
error = (vmodl.MethodFault) null,
result = ,
progress = <progress%>, 
reason = (vim.TaskReasonUser) {
dynamicType = ,
userName = "<user_name>",
},
queueTime = "YYYY-MM-DDThh:mm:ss.224526Z",
startTime = "YYYY-MM-DDThh:mm:ss.225526Z",
completeTime = ,
eventChainId = ####,
changeTag = ,
parentTaskKey = ,
rootTaskKey = ,
}

• • If the "cancelable" parameter in the above output is true, use below command to attempt canceling it:

vim-cmd vimsvc/task_cancel “<task_string>”

Example:

vim-cmd vimsvc/task_cancel “<haTask-<vm_id>-vim.VirtualMachine.<task_name>-####>”

 

• If the above methods dont succeed in cancelling the task(s), the task(s) can be cancelled by restarting the hostd service or rebooting the esxi host. However, do note there might be impact in case the environment contains vSAN, NSX or is connected to cloud like VMC on AWS. Please refer to the below article to learn about the same:

Restart Management Agents on Host