Consolidation of VM fails when Data Store runs out of space.

Products

• VMware vCenter Server

Issue/Introduction

• 3rd party backup software that uses snapshots is being used to backup vms
• Some VMS are set to ONLY backup the OS drive due to data churn rates on the DATA drive
• Someone adds a data drive to the VM, but forgets to exclude the new drive from the snapshot.
• Options to Consolidate the snapshots, power off the guest os, and other options are greyed out.
• Cancelling the Consolidation fails as well.

Environment

vSphere 8.0U3

Cause

Snapshot consolidation hangs due to the high data churn rate on the new drive that has had a snapshot done. Snapshots are designed for drives <1TB and with low data churn rates. 

Resolution

1. SSH into the host the VM is running on.
2. Get the vmid of the vm that is having issues consolidating:
   vim-cmd vmsvc/getallvms
   
   Output is similar to:
   
Vmid Name File Guest OS Version Annotation
1 vm1 [datastore1] vm1/vm1.vmx windows7Server64Guest vmx-08
3 testvm [iscsi1] testvm/testvm.vmx winNetDatacenterGuest vmx-08
   
   
3. vim-cmd vmsvc/get.tasklist VMID
   
   Output is similar to:
   
   (ManagedObjectReference) [
'vim.Task:haTask-112-vim.VirtualMachine.createSnapshot-3887'
]

4. Make note of the task identifier. In the example above, the task identifier is:
   
   haTask-112-vim.VirtualMachine.createSnapshot-3887

5. To get information about the status of a particular task, run the command:
   
   vim-cmd vimsvc/task_info task_identifier
   
   Where task_identifier is the string recorded in step 3.
   
   The output is similar to:
   
   (vim.TaskInfo) {
dynamicType = <unset>,
key = "haTask-112-vim.VirtualMachine.createSnapshot-3887",
task = 'vim.Task:haTask-112-vim.VirtualMachine.createSnapshot-3887',
description = (vmodl.LocalizableMessage) null,
name = "vim.VirtualMachine.createSnapshot",
descriptionId = "VirtualMachine.createSnapshot",
entity = 'vim.VirtualMachine:112',
entityName = "deploy-test",
state = "running",
cancelled = false,
cancelable = false,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 0,
reason = (vim.TaskReasonUser) {
dynamicType = <unset>,
userName = "root",
},
queueTime = "YYYY-MM-DDTHH:MM:SS.xxxZ",
startTime = "YYYY-MM-DDTHH:MM:SS.xxxZ",
completeTime = <unset>,
eventChainId = 2936866,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
}


6. If the status shows running, and the progress is static (not changing for very long periods of time) then the consolidation may be hung.
   
   
7. At this point, you should verify the vmdk sizes of the Data disks, and see if they are static or changing. IF they are changing frequently, then you have a high data churn rate. Consult with your IT department on how best to get the vm into a state of low data churn. 
   
   
8. At that point, alternative methods for backing up the data should be taken, (Data base dumps, agent assisted backup, etc) and backups verified to prevent data loss.
9. From within the GuestOS, shut down the VM.
10. Check the consolidation status in vCenter Task manager. It may continue consolidating. If it still appears hung after an hour, cancel the consolidation.
11. After the consolidation cancels. Perform a new consolidation task. For large Datasets this can take several hours to days, but the percentage mark should move a bit on the consolidation. 
    
    
12. Once consolidation is complete, make sure the backup software is set to NOT snapshot the Data drive, and only snapshot the OS. 
    
    
13. Power up the VM.

Additional Information

To find which host a vm is on:

In the vCenter UI, select the VM in the left hand pane, then on the right, check the Summary tab for the VM. Under the box  called "Related Objects", you will see the cluster, host and other things about the VM.