Snapshot removal task stops at 99%
search cancel

Snapshot removal task stops at 99%

book

Article ID: 341355

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps to monitor snapshot removal on ESXi using the watch command.


Symptoms:
  • The snapshot removal task hang at 95% or 99%. 
  • You cannot observe the progress when you delete a snapshot.
     
Note: For additional symptoms, see the Additional Information section.


Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

Note: Time involved to commit snapshots is environmental and subjective.

To estimate the time it will take to remove snapshots, see Estimate the time required to consolidate virtual machine snapshots

The consolidation/Removal task hanging at 99% is in most cases related to:
  • Virtual Machine snapshot size on disk (Delta or sesparse file size).
  • Storage performance.
  • Network connection interrupted between ESXI host and vCenter.
  • Size of the Virtual Machine.
The virtual machine may become unresponsive due to the stun process integrated with the Consolidation process. The stun/freeze happens to accommodate the changes to be written back to the base disk from the Delta disks.  At times if the virtual machine is generating a lot of I/O or the underlying storage is experiencing latency, the stun time can increase.

Resolution

For 5.x, see Monitor snapshot deletion using esxtop command (2146232)
Monitor using vim-cmd in ESXi

Finding consolidation tasks that are at 99%

  1. Log in as root to the ESXi host using SSH.
  2. Navigate to the virtual machine directory containing vmdk virtual disk files. Run the cd command to change the current directory.

    For example:

    cd /vmfs/volumes/Datastore_name/Virtual_Machine_name/
 
  1. Run less testdev02.vmx | grep vmdk to list vmdk (disks) of the VM in the vmx (configuration file of VM):
For example:
less testdev02.vmx |grep vmdk
scsi0:0.fileName = "testdev02.vmdk"
scsi0:1.fileName = "testdev02_1-000003.vmdk"
scsi0:2.fileName = "testdev02_2-000002.vmdk"
  1.  Run this command to list the snapshot files in the directory:
  • less testdev02.vmx | grep *sesparse.vmdk
  • For VMFS5 volume: less testdev02.vmx | grep *delta.vmdk
Listing the snapshot of the VM. 
-rw-------    1 root     root       18.2G Oct 29 10:04 testdev02_2-000001-sesparse.vmdk
-rw-------    1 root     root      943.6G Oct 29 10:09 testdev02_1-000002-sesparse.vmdk
-rw-------    1 root     root        1.8T Oct 29 10:09 testdev02_1-000001-sesparse.vmdk
-rw-------    1 root     root        1.7G Oct 29 20:03 testdev02_2-000002-sesparse.vmdk
-rw-------    1 root     root        3.1T Oct 29 20:03 testdev02_1-000003-sesparse.vmdk

Note: Sesparse is a default format for all delta disks on the VMFS6 datastores. On VMFS5, Sesparse is used for virtual disks of the size 2 TB and larger.
  1. In this example we have snapshots approximately 6 TB and this could be the main reason for the progress of the task. The process will continue to show 99% until the snapshot removal task completes and VM is using the flat files of the VMDK.
  2. Find the consolidation task with this command: vim-cmd vimsvc/task_list
For example:
[root@ESXi:~] vim-cmd vimsvc/task_list
(ManagedObjectReference) [
   'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
   'vim.Task:haTask-ha-host-vim.host.ImageConfigManager.installDate-321689752',
   'vim.Task:haTask-ha-host-vim.host.ServiceSystem.start-321689774',
   'vim.Task:haTask-ha-host-vim.host.ServiceSystem.refresh-321689779'
]
  1. View the details of the consolidation with this command: vim-cmd vimsvc/task_info <task_name_from_step_6>
[root@ESXi:~] vim-cmd vimsvc/task_info haTask-91-vim.vm.Snapshot.remove-321615207
(vim.TaskInfo) {
   key = "haTask-91-vim.vm.Snapshot.remove-321615207",
   task = 'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
   description = (vmodl.LocalizableMessage) null,
   name = "vim.vm.Snapshot.remove",
   descriptionId = "vm.Snapshot.remove",
   entity = 'vim.VirtualMachine:81',
   entityName = "TESTDEV",
   locked = <unset>,
   state = "running",
   cancelled = false,
   cancelable = true,
   error = (vmodl.MethodFault) null,
   result = <unset>,
   progress = 99,
   reason = (vim.TaskReasonUser) {
      userName = "vpxuser"
   },
   queueTime = "2019-10-29T10:08:21.403596Z",
   startTime = "2019-10-29T10:08:21.404185Z",
   completeTime = <unset>,
   eventChainId = -1078812089,
   changeTag = <unset>,
   parentTaskKey = <unset>,
   rootTaskKey = <unset>,
   activationId = <unset>
}
 

Monitor using watch in ESXi 6.x/7.x

To monitor the VMDK snapshot and base disks which are currently being updated, run this watch command:

This command refreshes the file referred by the on going task every 2 seconds. 

#   watch -d 'ls -luth | grep -E "delta|flat|sesparse"' 

where:

-d highlights the differences between successive updates

In addition, run this command to monitor the time stamp update of the base disks to confirm if the process is working. The result lists the files by their modification date.

ls -lrth |grep -E "flat|delta|sesparse"


Notes:
  • In ESX/ESXi 4.0 Update 2 and later, the process works differently in that the data in snapshots (deltas) are written directly to the base disk (flat).
  • If there are more than 10 snapshots, run this command to monitor the snapshot commit process and to prevent the screen from filling with many files:

    #while true;do date;ls -lht *vmdk|head -10;echo ________;sleep 3;done
     
  • You can quit the consolidation process monitoring by pressing Ctrl + C.


Additional Information

Powered by Wolken Software