Force stop vSphere Replication on vSAN datastore and retain existing VMDKs/Seeds for the Virtual machine
search cancel

Force stop vSphere Replication on vSAN datastore and retain existing VMDKs/Seeds for the Virtual machine

book

Article ID: 327067

calendar_today

Updated On:

Products

VMware Live Recovery VMware vSAN VMware vSphere ESXi

Issue/Introduction

  • Traditional method using the VMFS  datastore was to rename the folder at the destination VMFS datastore and then force stop replication , re-name the folder back to original and replicate the VM using existing Seeds 
  • We Cannot Re-Name the folders on the destination vSANdatastore (Object Store file system) since the VM folder name is symlink to namespace object UUID .
  • To safely force stop vsphere Replication and reconfigure replication once again , this KB covers the steps  to re-replicate the VM using existing seeds to a vSAN datastore on the destination.


Symptoms:
  • vSphere -Replication for a virtual machine is broken  The destination vm folder is located on a vSAN datastore. 
  • The Admin would like to stop/re-configure replication using existing seeds / vmdks to avoid a full sync to the destination vSAN datastore .


Environment

VMware vSAN 6.x
VMware vSphere Replication 6.x
VMware vSphere Replication 6.0.x
VMware vSAN 6.7.x
VMware vSphere Replication 6.5.x
VMware vSphere Replication 8.x
VMware vSAN 6.6.x
VMware vSAN 6.5.x

Cause

  • Vsphere Replication is in error state for a VM replicating to a vSAN datastore
  • Replication is in a RPO violation state . Force resync is failing for the VM running on vSAN datastore
  • RDID vdisk on the destination folder is corrupted , force re-configure and other attempts did not help sync up the VM .
  • Re-sync replication to the target vSAN datastore gets stuck at full sync for very long time and never progresses .

Resolution

Please follow the below steps carefully and make sure every step is documented and noted .
  • Note down the path / namespace object for the vm-folder where the data is being replicated to from the source virtual machine  and the files and object in that folder .
  • SSH to any of the hosts on the destination site/cluster which has access to the VM-namespace folder and view the list of files available .
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c] cd Test-vr-rep
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/c579a05b-4d12-f70f-df1c-005056b82734] ls -la
total 792608
drwxr-xr-t    1 root     root          2940 Sep 18 05:42 .
drwxr-xr-x    1 root     root           512 Sep 18 05:53 ..
-rw-------    1 root     root             0 Sep 18 04:25 .207ea05b-d5d7-dd9c-1a66-005056b8688c.lck
-rw-------    1 root     root             0 Sep 18 05:42 .2b90a05b-cbd0-7a1c-610b-005056b82734.lck
-rw-------    1 root     root             0 Sep 18 04:25 .337ea05b-aaf4-ae0b-073c-005056b8688c.lck
-r--------    1 root     root       1441792 Sep 18 04:06 .fbb.sf
-r--------    1 root     root     267026432 Sep 18 04:06 .fdc.sf
-r--------    1 root     root       1179648 Sep 18 04:06 .pb2.sf
-r--------    1 root     root     268435456 Sep 18 04:06 .pbc.sf
-r--------    1 root     root     262733824 Sep 18 04:06 .sbc.sf
drwx------    1 root     root           280 Sep 18 04:06 .sdd.sf
-r--------    1 root     root       4194304 Sep 18 04:06 .vh.sf
-rw-------    1 root     root           521 Sep 18 04:27 Test-vr-rep.vmdk
-rw-------    1 root     root          8684 Sep 18 04:32 hbrcfg.GID-2e816e27-c516-4761-bec4-6e7ec5e576b6.10.nvram.14
-rw-------    1 root     root          3151 Sep 18 04:32 hbrcfg.GID-2e816e27-c516-4761-bec4-6e7ec5e576b6.10.vmx.13
-rw-------    1 root     root          8684 Sep 18 04:27 hbrcfg.GID-cf9f2410-3997-47d0-8033-333de72398c6.9.nvram.12
-rw-------    1 root     root          3153 Sep 18 04:27 hbrcfg.GID-cf9f2410-3997-47d0-8033-333de72398c6.9.vmx.11
-rw-------    1 root     root           339 Sep 18 05:42 hbrdisk.RDID-b968fb4a-13c7-415c-a43d-8ef040fc1051.31.124727821486618.vmdk
-rw-------    1 root     root           339 Sep 18 04:27 hbrdisk.RDID-e219e8d8-b7a7-4ee0-8311-2ebba2323659.15.255651935167219.vmdk
-rw-------    1 root     root          3064 Sep 18 05:42 hbrgrp.GID-2e816e27-c516-4761-bec4-6e7ec5e576b6.txt
-rw-------    1 root     root          3064 Sep 18 04:27 hbrgrp.GID-cf9f2410-3997-47d0-8033-333de72398c6.txt
 
  • Create a new Folder (namespace)by browsing vSAN datastore (GUI) or by command line as Below :
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c] /usr/lib/vmware/osfs/bin/osfs-mkdir Dummy-Folder
0194a05b-5c07-0c48-c4f8-005056b82734
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c] cd 0194a05b-5c07-0c48-c4f8-005056b82734
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/0194a05b-5c07-0c48-c4f8-005056b82734]
  • Move all the content in the original folder to the new Test/Dummy folder , here in this example , Original folder "Test-vr-rep" (/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/c579a05b-4d12-f70f-df1c-005056b82734) to Dummy-Folder (/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/0194a05b-5c07-0c48-c4f8-005056b82734) . Please Double check and Confirm that there are no vmdk descriptors leftover on the source folder before moving to the next step .
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/c579a05b-4d12-f70f-df1c-005056b82734] mv * /vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/0194a05b-5c07-0c48-c4f8-005056b82734/
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/c579a05b-4d12-f70f-df1c-005056b82734] cd /vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/0194a05b-5c07-0c48-c4f8-005056b82734/
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/0194a05b-5c07-0c48-c4f8-005056b82734] ls -la
total 792608
drwxr-xr-t    1 root     root          2520 Sep 18 06:49 .
drwxr-xr-x    1 root     root           512 Sep 18 06:50 ..
-r--------    1 root     root       1441792 Sep 18 05:58 .fbb.sf
-r--------    1 root     root     267026432 Sep 18 05:58 .fdc.sf
-r--------    1 root     root       1179648 Sep 18 05:58 .pb2.sf
-r--------    1 root     root     268435456 Sep 18 05:58 .pbc.sf
-r--------    1 root     root     262733824 Sep 18 05:58 .sbc.sf
drwx------    1 root     root           280 Sep 18 05:58 .sdd.sf
-r--------    1 root     root       4194304 Sep 18 05:58 .vh.sf
-rw-------    1 root     root           521 Sep 18 04:27 Test-vr-rep.vmdk
-rw-------    1 root     root          8684 Sep 18 04:32 hbrcfg.GID-2e816e27-c516-4761-bec4-6e7ec5e576b6.10.nvram.14
-rw-------    1 root     root          3151 Sep 18 04:32 hbrcfg.GID-2e816e27-c516-4761-bec4-6e7ec5e576b6.10.vmx.13
-rw-------    1 root     root          8684 Sep 18 04:27 hbrcfg.GID-cf9f2410-3997-47d0-8033-333de72398c6.9.nvram.12
-rw-------    1 root     root          3153 Sep 18 04:27 hbrcfg.GID-cf9f2410-3997-47d0-8033-333de72398c6.9.vmx.11
-rw-------    1 root     root           339 Sep 18 05:42 hbrdisk.RDID-b968fb4a-13c7-415c-a43d-8ef040fc1051.31.124727821486618.vmdk
-rw-------    1 root     root           339 Sep 18 04:27 hbrdisk.RDID-e219e8d8-b7a7-4ee0-8311-2ebba2323659.15.255651935167219.vmdk
-rw-------    1 root     root          3064 Sep 18 05:42 hbrgrp.GID-2e816e27-c516-4761-bec4-6e7ec5e576b6.txt
-rw-------    1 root     root          3064 Sep 18 04:27 hbrgrp.GID-cf9f2410-3997-47d0-8033-333de72398c6.txt
[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/0194a05b-5c07-0c48-c4f8-005056b82734]

**NOTE ORIGINAL FOLDER WILL STILL CONTAIN lck files , which can be ignored :

[root@zh4:] cd /vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/Test-vr-rep

[root@zh4:/vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/c579a05b-4d12-f70f-df1c-005056b82734] ls -la
total 790528
drwxr-xr-t    1 root     root          1680 Sep 18 06:49 .
drwxr-xr-x    1 root     root           512 Sep 18 06:51 ..
-rw-------    1 root     root             0 Sep 18 04:25 .207ea05b-d5d7-dd9c-1a66-005056b8688c.lck
-rw-------    1 root     root             0 Sep 18 05:42 .2b90a05b-cbd0-7a1c-610b-005056b82734.lck
-rw-------    1 root     root             0 Sep 18 04:25 .337ea05b-aaf4-ae0b-073c-005056b8688c.lck
-r--------    1 root     root       1441792 Sep 18 04:06 .fbb.sf
-r--------    1 root     root     267026432 Sep 18 04:06 .fdc.sf
-r--------    1 root     root       1179648 Sep 18 04:06 .pb2.sf
-r--------    1 root     root     268435456 Sep 18 04:06 .pbc.sf
-r--------    1 root     root     262733824 Sep 18 04:06 .sbc.sf
drwx------    1 root     root           280 Sep 18 04:06 .sdd.sf
-r--------    1 root     root       4194304 Sep 18 04:06 .vh.sf
 
  • Head back to webclient , and execute FORCE Stop Replication for the virtual machine in question from both Source VCenter (Outgoing Replication) and Destination VC (Incoming Replication) .  Please  note that the VC will still try to delete the vmdk on the destination (Original folder) and fail the task at this stage .

  • Copy the vmdk files (all the base disks) back to the source folder and leave the rest on the dummy-folder
cp Test-vr-rep.vmdk /vmfs/volumes/vsan:5221428b209bc448-116434ddeaf3473c/c579a05b-4d12-f70f-df1c-005056b82734/
  • ​Complete Re-Configure Replication task , from the source VM and use existing seed at the destination , Replication using existing VMDK should finish successfully .








Workaround:
N/A

Additional Information

https://docs.vmware.com/en/vSphere-Replication/6.0/com.vmware.vsphere.replication-admin.doc/GUID-C7CD1006-7E2F-42B3-A3EC-429F365140E1.html

Impact/Risks:

If this procedure was not followed in correct order , the customer will lose the existing VMDK at the target site and a full sync to a new VMDK/Object  is needed to get the replication back to OK status.

The steps mentioned in the article should be executed by experts only. In case of doubts please open support ticket with VMware GSS with details of the environment.