Error : "Detected existing metadata in cluster <Cluster_MOID>. Run recovery workflow to ensure metadata is in sync" observed on creating Protection Group in vSAN Data Protection

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Error observed while creating new protection group:

"Detected existing metadata in cluster <Cluster_MOID>. Run recovery workflow to ensure metadata is in sync."

• Issue occurs after re-deploying the VLR appliance.

Environment

VMware Live Recovery 9.0.4

Cause

This issue occurs when redeployed VLR appliance/s fail to sync with vSAN Cluster metadata for previously created Protection Groups.

Resolution

Follow below steps to resolve the issue:

• Take snapshot of VLR appliance before we proceed with below steps.
  
  
• Copy Cluster DB from vSAN Datastore to newly deployed VLR appliance
  
  
• Recover appliance from cluster DB

# [root@:/vmfs/volumes/vsan:XXXXXXXXXXX] ls -ltr
   -rw-r--r--    1 root     root       snapstore.db

SSH to VLR appliance and copy above stated cluster DB to following location:

(Note : Create this path if it does not exist)

# /opt/vmware/db/snapservice/recovery/

Update folder permission created under /opt/vmware/snapservice/data/recovery/

# chmod -R 777 /opt/vmware/db/snapservice/recovery/

• Recover appliance from cluster DB

Locate recovery script recovery.py at location -- /opt/vmware/snapservice/api_scripts

• Perform Recovery of VLR appliance using following command:

# python recovery.py -c 'domain-XX' --vcUser '[email protected]' -f /opt/vmware/db/snapservice/recovery/snapstore.db

(-c is cluster MoID of cluster for which the Protection Group creation fails. This is the same MoID that is observed in the error message above. 
Select the cluster in vSphere Client and check URL to get Moid that is in format --> 'domain-cXX'.)

On successful execution of above command, the output would look something like this:

Please enter the vCenter password:
VC IP: <vcenter_ip_or_fqdn>:443
Invoking recovery API
API response : <Response [202]>
Waiting for task to complete

{'cancelable': False, "description': {'args': [], 'default_message': 'Cluster DB recovery task', 'id': 'com.vnware.vsan.snapservice.RecoverDB'}, 'end_time': 'YYYY-MM-DDTHH:MM:SS.MSZ', "operation':
'recover', 'progress': {'completed': 0, 'message': {'args': [], 'default_message': 'Cluster DB recovery task', 'id': 'com.vmware.vsan.snapservice.RecoverDB'}, 'total'; 180}, 'result': None, 'service': 'com.vmware.snapservice_internal.clusters', 'start_time': 'YYYY-MM-DDTHH:MM:SS.MSZ', 'status': 'SUCCEEDED", 'target': {'id': "domain-cXX', 'type': 'ClusterComputeResource'}, "user': '[email protected]'}

 

• Repeat this process for the other site VLR appliance.
  
  
• Create new Protection Group:
  
  -- vSphere Client > vSAN Cluster > Configure > vSAN > Data Protection > Create Protection Group