Consolidating snapshots in ESX/ESXi 3.x and 4.x
search cancel

Consolidating snapshots in ESX/ESXi 3.x and 4.x

book

Article ID: 304464

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Virtual machines with snapshots or delta disks stop responding or fail during a storage migration.
  • When powering on a virtual machine, you see this error message:

    The parent virtual disk has been modified since the child was created.
     
     
  • A virtual machine unexpectedly powers off and you see this error when you attempt to power it back on:

    A general system error occurred: internal error.
     
     
  • Virtual machine log contains the entry:

    vcpu-0| Msg_Question: msg.hbacommon.outofspace reply=0
    vcpu-0| Msg_Question:
    vcpu-0| msg.hbacommon.outofspace There is no more space for the redo log of W2003Sbs1-000001.vmdk.
    vcpu-0| You may be able to continue this session by freeing disk space on
    vcpu-0| the relevant partition, and clicking Retry. Otherwise,
    vcpu-0| click Abort to terminate this session.
     
     
  • You are unable to remove snapshots for virtual machines on one or more datastores.


Environment

VMware ESXi 3.5.x Embedded
VMware ESX Server 3.0.x
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESX Server 3.5.x
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX 4.1.x

Resolution

Note: This article addresses ESX/ESXi 3.x and 4.x. For information on vSphere 5.x, see Consolidating snapshots in vSphere 5 (2003638).


This error can occur if:
  • The base disk of the virtual machine has been modified after a virtual machine snapshot or delta disk has been created.
  • You have run out of space on a datastore containing the snapshot or delta disk.
  • The guest operating system suffered an exception or kernel panic when a virtual machine snapshot was taken.

For more details about snapshots and recreating missing virtual machine descriptor files, see: Cannot power on a virtual machine because the virtual disk cannot be opened (1004232).

Overview

A VMware ESX virtual machine disk consists of a descriptor (ends in .vmdk) and an extent (ends in -flat.vmdk). When a virtual machine has been snapshotted, the attached virtual hard disks (otherwise known as the base disks) no longer receive guest OS modifications or writes; they are stored in separate virtual machine delta disk files (ending in -delta.vmdk). There is one delta file per base disk that is snapshotted.
 
With exception to thin-provisioned virtual machine disks, virtual machine disks typically have a reserved or set space requirement on a datastore. Snapshots, however, require additional space and consideration. They increase in size as further modifications or writes are stored.
 
For example:
  1. If 10GB of changes were completed on one snapshotted virtual machine, its delta disk file size will increase proportionally by 10GB.
     
  2. Creating another snapshot causes the existing snapshot delta disk to retain its current size, however the next delta disk will begin to store all forthcoming modifications and writes. The base disk is still left unmodified since inception of the first snapshot.
  3. If another 15GB of changes were performed by the guest operating system, a total of 25GB of snapshot delta has now been recorded over the respective virtual machine's two snapshot delta disk files.
     
  4. For versions prior to VMware ESX 4.0 Update-2, the task of consolidating all snapshots (Remove All Snapshots task) causes unique changes stored only in the second snapshot delta disk to be copied upwards through the snapshot chain and into the first snapshot, or its "parent."
  5. This effect is recursive for each preceding parent file. As a result, the first snapshot delta disk file will grow by up to 15GB, accommodating all new blocks. Any common changes stored in both snapshot levels does not require additional space, however.
     
  6. The end result is a datastore requiring 40GB, or 25GB + 15GB.

Notes:

  • Additional space is required if the virtual machine is running during a Remove All Snapshots task. The amount of space consumed is dependent the amount of snapshot delta information the virtual machine has to temporarily store while its consolidation process completes.
  • If the same 10GB were changed in the second snapshot, and just 5GB of unique block changes occurred, then the first snapshot will increase by 5GB during the consolidation, not the complete 15GB.
  • Opting to save virtual machine memory contents with each Create Snapshot task requires additional space from the VMFS datastore. The amount of space required is equivalent to the amount of memory assigned to the virtual machine. This requirement can apply multiple times over, if multiple snapshots including virtual machine memory contents are created.
If a datastore fills up due to delta files accumulating too much space, or the datastore runs out of space during a snapshot consolidation effort, the symptoms outlined previously will occur. Migration or removal of stored data and snapshot consolidation will be required to resolve the issue.
 
For additional information on virtual machine snapshots, see Understanding virtual machine snapshots in VMware ESXi and ESX (1015180).
 

VMware ESX 4.0 Update 2 or later

VMware ESX now incorporates improved consolidation procedures, which lessens the demand of free space. You are able to consolidate virtual machine delta disks even while minimal free space on your datastore is available.

To consolidate the snapshots for a virtual machine currently registered to an ESX 4.0 Update 2 or later host:
  1. Power off the virtual machine.
  2. Open the virtual machine's Snapshot Manager.
  3. Remove one or more snapshots by click Delete or Delete All.
  4. Allow time for the consolidation to complete. Larger snapshots and delta disks will take more time to complete.

VMware ESX 4.0 Update 1 or earlier

Cloning the affected virtual machine or specific virtual disk to another location preserves the original files, and consolidates the disk's snapshot delta information with a copy of the base disk into the new virtual machine disk.
 
Outright snapshot removal or consolidation is an irreversible process. This process is possible from the VMware vSphere and Infrastructure Client, as well as through the vSphere Management Assistant (vMA), vSphere CLI, Power CLI, SDK/APIs, and the ESX host terminal.
 
Note: Virtual machine clones are possible through a client connection to VMware vCenter and VirtualCenter. If you do not have either of these products, virtual machine cloning is not available. You may, however, still clone and consolidate individual virtual machine disks.
 

Solution Prerequisites

  • Review the virtual machine's configuration and identify the locations and filenames of the attached virtual disks.
  • Verify what specific delta virtual disk(s) the virtual machine is referencing. You can determine this either by reviewing the virtual machine hardware configuration, highlighting a virtual disk, and noting each disk's file name, or you can review the virtual machine's configuration ( .vmx) file.
  • Also note the size of each virtual machine hard disk (specifically the base disk files). Destination clones will be of these determined sizes.
  • Cloning of virtual disks may be required as part of your effort towards consolidation and reclaiming of space. Verify that you have enough space on one or more datastores to receive a consolidated clone of the virtual machine in entirety or in part (a virtual disk).
  • Perform a sanity check of the virtual machine disk tree, verifying Content ID (CID) values between multiple snapshot delta disk files, and other virtual machine disk descriptor values. For more important information on this topic, see: Resolving "The parent virtual disk has been modified since the child was created" CID mismatch errors (1007969).
  • Conduct a review of datastore space consumption and assess what datastores disk or virtual machine clones can be saved to, as part of a consolidation effort.

Solutions

These solutions ensure that the original disk files are not modified. The consolidated clone can be reviewed before making a decision to allow the clone copy to take over services for the affected virtual machine.

Note: Third party backup software may fail to back up a virtual machine if there are orphaned delta or ctk files in the VM directory with an error about missing ctk.vmdk files. This occurs as the snapshot or change tracking files cannot be created because they are already in use. To work around this issue, rename or remove the orphaned .delta.vmdk or .ctk file(s).

 

Additional Information

For other related information pertaining to resolving snapshot issues, see:


After understanding the snapshot or delta disk layout, you may begin to clone disks, as required. The information collected above provides the ability to roll back changes, wherever required, as well as necessary details for the cleanup (deletion) process.
 
Migrating the cloned disks back to the virtual machine's working directory
 
Once you have cloned the virtual machine disk and its delta information to a single consolidated disk in another location, you may wish to relocate the disk clone back to the previous datastore location where the virtual machine resides.

This is possible using vmkfstools provided in the ESX host terminal, vMA, and vCLI. The -E switch will move the disk, and the -i switch will clone it. We recommend you to use -i option as this will preserve the source disk. For more information, see Cloning and converting virtual machine disks with vmkfstools (1028042)

When you have selected a process, perform it in this order:
  1. Remove the virtual machine hard disk from the virtual machine's configuration.
  2. Complete your migration using the VMware vSphere or Infrastructure client's migrate functionality when connected to vCenter. Otherwise attempt this using the terminal, vMA, or vCLI with vmkfstools.
  3. When the migration completes, re-add the virtual machine hard disk to the virtual machine's configuration.
  4. Start the virtual machine.

Cleanup: Removing the original/old virtual machine disks via vMA and vCLI

Note: It is important that the virtual machine remains powered-on with its files locked throughout these steps. While powered on, deletion of in-use files (which are otherwise required) is not possible, in the event of a mistake being made anywhere in the deletion process. This procedure also guarantees an audit trail. Terminal access may not offer a complete command history.

  1. Confirm the virtual machine is running.
  2. List the virtual machine's datastore contents. Some or all of the snapshots should still be present.
     


    vifs --server -D '[Datastore Name] examplevm'
     
  3. For the virtual disk that was successfully cloned, delete its original delta and base disk files:
     


    vmkfstools --server -U '[Datastore Name] examplevm/examplevm-000003.vmdk'
    vmkfstools --server -U '[Datastore Name] examplevm/examplevm-000002.vmdk'
    vmkfstools --server -U '[Datastore Name] examplevm/examplevm-000001.vmdk'
    vmkfstools --server -U '[Datastore Name] examplevm/examplevm.vmdk'

    This error is displayed if you attempt to remove disk files that are locked or in-use. This otherwise may be indicative of attempting to remove the wrong disk files:


     
  4. Confirm that the disk files have been removed:

    vifs --server -D '[Datastore Name] examplevm'

 

Cleanup: Removing the base disk and related snapshot delta disks via the ESX host terminal

Note: See the section Confirming the Virtual Machine's Snapshot Delta Disk Layout Using the ESX Host Terminal below before proceeding with the provided steps.

  1. Confirm that the respective delta disk files are not locked while this virtual machine is powered-on. They should not be, at this stage. Run this command:

    touch -a examplevm-??????-delta.vmdk

    Note: Running the command with "??????" would test all delta files in the directory. If you have multiple virtual hard disks, substitute "examplevm-??????" with applicable values to the virtual disk in question (previously cloned).

    If the command is successful to all files, no output is given and you are returned to the prompt. If there are failures due to locking, however:

    touch: cannot touch `examplevm-000001-delta.vmdk': Device or resource busy
    touch: cannot touch `examplevm-000002-delta.vmdk': Device or resource busy
     
  2. Remove the snapshot delta and base disk extent/descriptor files using the vmkfstools command.

    vmkfstools -U examplevm-000003.vmdk
    vmkfstools -U examplevm-000002.vmdk
    vmkfstools -U examplevm-000001.vmdk
    vmkfstools -U examplevm.vmdk

    Note: The descriptor files are referenced, as opposed to the extents (*-delta.vmdk or *-flat.vmdk files)

    Again, any failures due to locked files will present an error:

    Failed to delete virtual disk: Device or resource busy (1048585).



Powered by Wolken Software