No more space for the redo log error when attempting to start a virtual machine

Products

• VMware vSphere ESXi

Issue/Introduction

When starting a virtual machine, a dialog appears stating there is no more space for the redo log , and prompts you to retry or abort.

Environment

VMware ESX 4.0.x
VMware ESX Server 3.0.x
VMware ESX Server 2.0.x
VMware ESX 4.1.x
VMware ESX Server 2.5.x
VMware ESX Server 2.1.x
VMware ESX Server 3.5.x

Resolution

The datastore in which the virtual machine resides in is out of disk space. The following are steps used to commit all snapshots and attain additional disk space.

Note: There may be corruption on the last snapshot if the virtual machine was abruptly powered down due to lack of disk space.

Identify any virtual machine with snapshots by running the following command at the service console:

1. Log into the console of the ESX host as root.
2. Run the following command:
   
   find /vmfs/volumes -name *-0000[0-9][0-9]*.vmdk
   
   
3. Make note of all the file locations and the virtual machine names which include delta VMDK files.

All snapshots must be successfully committed to free up any disk space within the datastore. The snapshot commit process can be done in one of several ways.

Commit via the Virtual Infrastructure Client

1. Right-click on the identified virtual machine.
2. Click Snapshot > Snapshot Manager.
3. Click Delete All to commit all of the snapshots to the parent disk.

Commit via the Command Line

1. Log into the Service Console of the ESX host hosting the identified virtual machine as root.
2. Run the following command to list the virtual machines being hosted by this ESX host:
   
   vmware-cmd -l
   
   
3. Run the following command to commit all snapshots of the virtual machine:
   
   vmware-cmd <full path to vmx> removesnapshots

Note: There is no way to predetermine how long the commit process takes. If the size of the snapshot files are large, the commit process may exceed the default timeout of the VI Client, and a timeout message may be reported. Despite this timeout message, the commit process continues with no progress bar or status indicator. There is no way to predetermine the amount of time it takes to commit the snapshot as it depends too many factors which include, but are not limited to, snapshot size, array performance, HBA performance, fiber/network performance, and load.

In the event that space is an issue, there are certain operations which can be conducted to free up space for the purposes of committing the snapshots. The operations include:

1. Powering off the virtual machine
   
   This operation deallocates the swap file. A virtual machine's swap file is usually the same size as the allocated amount of RAM. This operation does not have to be done to the machine which is running off of the snapshot. If the issue is affecting a production virtual machine, non-production machines residing in the same datastore can be powered off to free up storage for the commit operation. For more information, see http://www.vmware.com/pdf/esx3_memory.pdf .
   
   
2. Add an extent to the existing datastore
   
   If there is a lack of disk space, the Add Extent wizard can be used to increase the amount of space available to this datastore. The Add Extent operation is irreversible, and creates a dependence of multiple LUNs for a single datastore. For more information, see http://www.vmware.com/pdf/vmfs-best-practices-wp.pdf .
   
   
3. Clone the virtual machine to a datastore that has more space
   
   To clone the virtual machine to a datastore that has more space:
   
   
   1. Right-click on the virtual machine which had been identified.
   2. Select Clone.
   3. Go through the clone wizard and select a datastore with adequate space.
   4. Verify that the clone has no snapshots.
   5. Start up the clone of the original virtual machine and verify that it is functioning in the same capacity as the original virtual machine.