Reprotect operation fails with the error "Unable to reverse replication for the virtual machine 'xxxxx'. An error occurred while deleting a snapshot"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms

• Replications are configured using vSphere Replication

• After performing a planned migration, when trying to reverse the direction of replication using reprotect, operation fails with the below error message at "Configure storage to reverse direction" phase 

  Unable to reverse replication for the virtual machine 'xxxxxx'. An error occurred while deleting a snapshot: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.

Environment

vSphere Replication 9.x

Cause

This issue is observed due to the presence of stale/corrupt redo log files (hbrdisk.RDID) on the target datastore.

After the initial full synchronization is complete, vSphere Replication starts to track the changed blocks on the source site and periodically transfers them to the target site. This process is called an incremental sync. As a result of the incremental sync completion, vSphere Replication creates a redo log file (hbrdisk.RDID). The data in the redo log files will be consolidated to the base disk after the replication has completed successfully. If the redo log files are corrupt, the changes tracked in the redo log files cannot be consolidated into the base disk (seed disk) resulting in the snapshot deletion error.

Cause Justification

On browsing the target datastore where the replication was configured previously, we can see multiple hbrdisk.RDID files, indicating the changes were not committed to the base disk.

For instance, cd into the vm folder in the target datastore using the below commands

cd /vmfs/volumes/<Datastore_name>/<vm_name>

On listing the contents of the folder (ls -ltrh), you will find multiple hbrdisk.RDID files along with vmsn and vmsd files used for storing the virtual machine snapshot information

-rw-------    1 root     root         412 Mar  5  2024 hbrdisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.xxxxx.xxxxxxxxxxxxxx.vmdk
-rw-------    1 root     root         412 Mar  6  2024 hbrdisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.xxxxx.xxxxxxxxxxxxxx.vmdk
-rw-------    1 root     root         411 Mar  7  2024 hbrdisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.xxxxx.xxxxxxxxxxxxxx.vmdk
-rw-------    1 root     root         337 Mar 23  2024 hbr-persistent-state-RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.psf
-rw-------    1 root     root         572 Mar 27  2024 hbrimagedisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.vmdk
-rw-------    1 root     root       18.8M Apr  2  2024 hbrimagedisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-ctk.vmdk
-rw-------    1 root     root      635.9K Apr  2  2024 vmware-2.log
-rw-r--r--    1 root     root        8.0K Apr  2  2024 vm-2.scoreboard
-rw-------    1 root     root       28.5K Aug  3  2024 xxxxxxxxxx-Snapshot8.vmsn
-rw-------    1 root     root       28.5K Aug  3  2024 xxxxxxxxxx-Snapshot7.vmsn
-rw-------    1 root     root       28.5K Aug  3  2024 xxxxxxxxxx-Snapshot6.vmsn
-rw-------    1 root     root        4.4K Aug  3  2024 xxxxxxxxxx.vmsd
-rw-------    1 root     root      212.8K Aug  3  2024 vmware-3.log
-rw-r--r--    1 root     root        8.0K Aug  3  2024 vm-3.scoreboard
-rw-------    1 root     root         572 Aug  9  2024 hbrimagedisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.vmdk
-rw-------    1 root     root       18.8M Aug  9  2024 hbrimagedisk.RDID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-ctk.vmdk
-rw-------    1 root     root      247.6K Aug  9  2024 vmware-4.log
-rw-r--r--    1 root     root        8.0K Aug  9  2024 vm.scoreboard
-rw-------    1 root     root         150 Aug  9  2024 hbrcfg.GID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.vmxf.1112
-rw-------    1 root     root        8.5K Aug  9  2024 hbrcfg.GID-89c1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.nvram.1113
-rw-------    1 root     root        8.5K Sep  6 13:57 xxxxxxxxxx.nvram
-rw-------    1 root     root         150 Sep 12 14:22 xxxxxxxxxx.vmxf
-rw-------    1 root     root      215.9K Sep 12 14:22 vmware.log
-rw-r--r--    1 root     root        7.2K Sep 12 14:22 xxxxxxxxxx.scoreboard
-rw-------    1 root     root       18.8M Sep 12 14:22 xxxxxxxxxx-ctk.vmdk
-rw-------    1 root     root        3.8K Sep 12 14:23 xxxxxxxxxx.vmx
-rw-------    1 root     root         766 Sep 12 14:24 xxxxxxxxxx.vmdk

Resolution

To resolve this issue, move all the hbrdisk.RDID files and vmsn and vmsd files to a different folder and rerun the reprotect operation.

• Create a folder inside the vm folder and move all the stale files
  mkdir backup
mv filename backup

• After moving the files, the target datastore should only have the seed disk details like below

   

  -rw-------    1 root     root      213.4K Mar  9  2024 vmware-1.log
-rw-r--r--    1 root     root        8.0K Mar  9  2024 vm-1.scoreboard
-rw-------    1 root     root      635.9K Apr  2  2024 vmware-2.log
-rw-r--r--    1 root     root        8.0K Apr  2  2024 vm-2.scoreboard
-rw-------    1 root     root      212.8K Aug  3  2024 vmware-3.log
-rw-r--r--    1 root     root        8.0K Aug  3  2024 vm-3.scoreboard
-rw-------    1 root     root      247.6K Aug  9  2024 vmware-4.log
-rw-r--r--    1 root     root        8.0K Aug  9  2024 vm.scoreboard
-rw-------    1 root     root        8.5K Sep  6 13:57 xxxxxxxxxx.nvram
-rw-------    1 root     root         150 Sep 12 14:22 xxxxxxxxxx.vmxf
-rw-------    1 root     root      215.9K Sep 12 14:22 vmware.log
-rw-r--r--    1 root     root        7.2K Sep 12 14:22 xxxxxxxxxx.scoreboard
-rw-------    1 root     root       18.8M Sep 12 14:22 xxxxxxxxxx-ctk.vmdk
-rw-------    1 root     root        3.8K Sep 12 14:23 xxxxxxxxxx.vmx
-rw-------    1 root     root         766 Sep 12 14:24 xxxxxxxxxx.vmdk
drwxr-xr-x    1 root     root       80.0K Oct 28 10:12 backup

• Rerun the reprotect operation