Error: Duplicated key not allowed when creating local vSAN Data Protection group after cluster creation / migration

Products

• VMware vSAN
• VMware Live Recovery

Issue/Introduction

When attempting to create a local Protection Group in vSAN Data Protection after migrating hosts to a new cluster, the task fails. This occurs even if prior snapshots and protection groups were removed before the migration.

UI Error seen after creating Protection Group: "Enabling local protection for workloads not on vSAN ESA datastore is not supported"

Errors seen in log : /var/log/vmware/snapservice/snap-service.log

• Failed to create clusters entry in DB","cluster":"domain-c######","error":"duplicated key not allowed"
"UPDATE \"clusters\" SET \"mo_id\"='domain-c#####',\"name\"='New Cluster name',\"uuid\"='########-####-####-####-############',\"vsan_mode\"='',\"vsan_personality\"=2 WHERE uuid = '########-####-####-####-############' OR mo_id = 'domain-c#####'","affected_rows":0,"elapsed":0.00044285,"error":"duplicated key not allowed"

Environment

• VCF Protection and Recovery 9.1.0

• VMware Live Recovery 9.0.4, 9.0.5

• VMware vSAN ESA 8.x

Cause

When The vSAN Data Protection database retains the vSAN ESA sub-cluster UUID mapping from the old cluster object, Snapshot Manager is unable to properly identify the vSAN Datatore as 'Valid'. When a new cluster entry is added (To the vCenter Server Database), it triggers a duplicate key violation because the UUID is still bound to the stale cluster entry if it still exists in vCenter inventory

Resolution

For Protection and Recovery 9.1.0:

1. Migrate hosts to the new cluster.

   For ESA cluster without encryption

   For ESA cluster with encryption

2. Once hosts are migrated and skyline health errors are resolved: Turn off vSAN on old cluster and then delete old cluster from vCenter

3. Delete the old cluster object from vCenter

4. Create the Protection Group

For Live Site Recovery 9.0.4 or 9.0.5 Please contact Broadcom support for assistance.

Additional Information

Important Note: All  vSAN Protection groups and snapshots on the VMs will need to be removed prior to migrating cluster and will need to be re-created as they are tied to the cluster ID and cannot be migrated at this time. 

The dp-monitoring service in Protection and Recovery 9.1.0 will inform snap-service of the cluster changes once old cluster is removed from vCenter. This service is not present in 9.0.4 and 9.0.5 and manual intervention will be required.