Datastore filling up with REDO files not being deleted by Veeam back up

Products

• VMware vCenter Server

Issue/Introduction

Datastore filling up with no changes or apparent snapshots on the vm's inside the datastore.

 

Environment

 ESXi 7.0.X and above 

Cause

Reviewing disks on the vm folder REDO log remains when snapshot has been deleted.

cat *.vmx | egrep -i vmdk
scsi0:1.fileName = "VMname.vmdk"
scsi0:0.fileName = "VMname.vmdk-xd-delta.vmdk"
scsi0:0.redo = "./VMname.vmdk-xd-delta.vmdk.REDO_tzcGyi"
 
cat ls_-isla-vmfsvolumes7dc6a2cd-898e8f72PMASE-2021c-01.txt
  
2062352501                408 -rw-------    1 root     root             413696 Sep 13 21:05 VMname-xd-delta-delta.vmdk
2062352549                  4 -rw-------    1 root     root                     525 Sep 13 21:05 VMname-xd-delta.vmdk
2062353241                12 -rw-------    1 root     root             6554112 Mar  3 00:00 VMname-xd-delta.vmdk-ctk.REDO_tzcGyi
2062353224    17552160 -rw-------    1 root     root     17918480384 Mar  5 19:37 VMname-xd-delta.vmdk-delta.REDO_tzcGyi
2062353232                  4 -rw-------    1 root     root                     531 Mar  3 00:00 VMname-xd-delta.vmdk.REDO_tzcGyi
2062361897                  4 -rw-------    1 root     root                   1536 Mar  3 00:00 VMname_IdentityDisk-ctk.vmdk
2062361888               56 -rw-------    1 root     root           16777216 Mar  5 08:22 VMname_IdentityDisk-flat.vmdk
2062361885                 4 -rw-------    1 root     root                     641 Mar  3 00:01 VMname_IdentityDisk.vmdk
2062352645                 0 -rw-------    1 root     root         152043520 Mar  3 02:16 vmx-PMASE-VMname.vswp 

Resolution

Reach out to Veeam to review why the REDO is not getting removed. 

Additional Information

1. Snapshot Redo Log:
Snapshot: When you take a snapshot of a VM, VMware creates a copy of the VM’s disk state at that moment. Any changes to the VM’s disk after the snapshot are written to a separate file, and this file is the delta file.
Delta Disk: A delta disk is a virtual disk file that stores all the changes made to the VM’s original disk after the snapshot was taken. It essentially "records" the modifications that occur while the snapshot exists.
Redo Log: The .REDO file is a special type of delta file used in the context of snapshots. It stores the changes made to the VM’s virtual disk while the snapshot was active.
The name xd-delta.vmdk.REDO suggests that it's a redo log associated with the VM's delta disk (e.g., xd-delta.vmdk). The REDO suffix indicates that this is the part of the snapshot where the changes (writes) are stored, and it will be merged back into the main disk file (e.g., xd.vmdk) once the snapshot is deleted or committed.

2. When Is the .REDO File Created?
The .REDO file is created when a snapshot is taken for a VM. The file accumulates changes made to the VM’s virtual disk after the snapshot.
It’s created alongside the delta file (.delta.vmdk) and is used to record the changes during the lifetime of the snapshot.

3. What Happens to the .REDO File?
When you delete the snapshot or merge it, the changes stored in the .REDO file are committed back into the original virtual disk (e.g., xd.vmdk).
If the snapshot is not deleted, the .REDO file will continue to grow, and the virtual machine’s disk will keep growing as new changes are written to it.

4. Understanding the Naming Convention:
xd-delta.vmdk: This is the delta file where changes to the virtual disk are stored while the snapshot is active.
xd-delta.vmdk.REDO: This file holds the redo logs, which are incremental changes that happened to the virtual disk after the snapshot was created.

5. Impact of .REDO File Growth:
If snapshots are kept for a long time, the .REDO files can grow significantly, which can consume large amounts of storage. It's important to regularly manage and delete snapshots to avoid running out of space and affecting performance.
It's best practice to consolidate and delete snapshots when they’re no longer needed to prevent these files from taking up unnecessary disk space.

6. How to Handle the .REDO File?
Check Snapshot Status: Ensure that there are no unnecessary snapshots. You can check and delete snapshots from the Snapshot Manager in vSphere.
Consolidate Snapshots: Sometimes, snapshots might not automatically consolidate when deleted. You can manually initiate snapshot consolidation through the VM options in vSphere or via SSH.
Monitor Disk Usage: Regularly check datastore usage to ensure that snapshot-related files like .REDO files aren’t causing storage problems.

7. Important Note:
If you're encountering an unusually large .REDO file, or if snapshots aren't being properly consolidated, it could indicate issues with snapshot management. You might need to consolidate the snapshots or remove orphaned ones to reclaim space.

In Summary:

The xd-delta.vmdk.REDO file is part of the snapshot mechanism in VMware.
It stores changes made to the virtual disk after a snapshot is taken.
The file grows as long as the snapshot exists and will be merged back into the main disk when the snapshot is deleted or consolidated.
Regularly managing and deleting snapshots is crucial to avoid bloated .REDO files and excessive disk usage.