OVF deployment or Virtual machine creation fails with Error "Datastore does not match current VM policy" on vSAN datastore

Products

• VMware vCenter Server

Issue/Introduction

• When deploying an OVF or creating a new virtual machine, an error occurs during storage selection on the Workload datastore: Datastore does not match current VM policy
• 

 

• From /var/log/vmware/sps.log :

  • YYYY-MM-DDThh-mm-ss.702-04:00 [pool-15-thread-2] DEBUG opId=mdqchjep-206542-auto-4fdc-h5:7006313 Mapping fault com. vmware.vim.binding. vim. fault. NotFound to storage common fault.

    YYYY-MM-DDThh-mm-ss.702-04:00 [pool-4-thread-11] DEBUG opId=mdqchjep-206542-auto-4fdc-h5:70063137 com. vmware. vim. sms. policy. PolicyManagerImpl - [queryMatchingContainer] Response: [MatchingContainerInfo [datastore=ManagedObjectReference: type = Datastore, value = datastore-####, serverGuid = null, error=NotFound [message=Container id is not present. ], info=null, groupId=null, resourceUsage=null, howMany=0, cacheControl=0]]

    YYYY-MM-DDThh-mm-ss.705-04:00 [pool-4-thread-11] DEBUG opId=mdqchjep-206542-auto-4fdc-h5:70063137 com. vmware.pbm. placement . impl. PlacementSolverImpl - [checkCompatibility] Result: [CompatibilityResult {hub = {hubId = datastore--####, hubType = Datastore}, matchingResources = null, howMany = null, utilization = null, warning = null, error = [ (pbm. fault. CompatibilityCheckFault) {
faultCause = null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl. LocalizableMessage) {
dynamicType = null,
dynamicProperty = null,
key = com. vmware. pbm.pbmFault. locale,
arg = (vmodl. KeyAnyValue) [
(vmodl. KeyAnyValue) {
dynamicType = null,
dynamicProperty = null,
key = summary,
value = Container id is not present.

• In vCenter Web Client navigate to vSAN cluster  and go to Monitor tab. Under monitor tab, under vSAN section, click on Support  and you should see Obfuscation highlighted.
• 

Note: The string of alphanumeric characters before the ':' and not highlighted is the Vc-Uuid and also aligns to the Config Generation in the vSAN cluster get output as seen below in the next step. The string of alphanumeric characters after the ':' is the vSAN cluster UUID and should align with the Sub-Cluster UUID as seen in the below vSAN cluster get output.

 

• SSH to one of the hosts in the cluster that is part of the vSAN cluster and run esxcli vsan cluster get and we can see the vSAN cluster UUIDs don't match up.
  • OUTPUT : vSub-Cluster UUID: 5250e5de-7689-0f8e-6896-############
• From VCDB,
  • Ran SELECT id,vsan_uuid FROM vpx_compute_resource; (to get a list of all the vSAN clusters.)
  • OUTPUT : 52417139-b5b4-e630-a8f8-###########
• Mismatch is observed between the vSAN Cluster UUID and the Sub-Cluster UUID in the ESXi host which is part of that vSAN cluster.

 

Environment

• VMware vCenter Server 8.x
• VMware vCenter Server 7.x
• VMware vSAN 7.x
• VMware vSAN 8.x

Cause

• A VM policy incompatibility may occur if the vSAN cluster UUID does not match the sub-cluster UUID on an ESXi host that is part of the vSAN cluster.

Resolution

Note : Please valid snapshot of the vCenter before proceeding with the below steps. (If vCenter is in linked mode, take offline snapshot of all vCenters in linked mode before proceeding.)

Caution: Please contact the Broadcom Support Team, before proceeding with the below steps

To fix this and get the vSAN cluster UUID in sync between vCenter and ESXi do the following:

1. SSH to the vCenter appliance and login as "root".

2. Stop the vpxd service by running service-control --stop vpxd

3. Stop the content-library service by running service-control --stop content-library

4. Run /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres to access the postgres database. Refer the instructions on KB Interacting with the vCenter Server Appliance 6.5/6.7/7.0 embedded vPostgres Database

5. Then run SELECT id,vsan_uuid FROM vpx_compute_resource; to get a list of all the vSAN clusters.

   VCDB=# SELECT id,vsan_uuid FROM vpx_compute_resource;
    id |              vsan_uuid
   ----+--------------------------------------
    21 | 52417139-b5b4-e630-a8f8-xxxxxxxxxxxx
   (1 row).

6. Then run "SELECT * FROM vpx_compute_resource_vsan_host WHERE cluster_uuid='<vSAN Cluster UUID as seen in vCenter Web Client>' order by host_id;" to see all the hosts associated with the cluster_uuid you want to view.

   • For example: SELECT * FROM vpx_compute_resource_vsan_host WHERE cluster_uuid='52417139-b5b4-e630-a8f8-xxxxxxxxxxxx' order by host_id;

7. Then run 'Update vpx_compute_resource SET vsan_uuid=<vSAN Cluster UUID as seen via ESXi>' WHERE id=<comp_res_id of the cluster you're working with>;' to correct the vSAN Cluster UUID.
   • For example: UPDATE vpx_compute_resource SET vsan_uuid = '5250e5de-7689-0f8e-6896-############' WHERE id=2030;
8. Restart all vCenter services by running service-control --stop --all && service-control --start --al
9. Then from vSAN cluster from vCenter > Cluster > Monitor > vSAN > Skyline Health , select the Alert "vCenter is not authoritative" > Click on "Troubleshoot" > Then Click the "Remediate Inconsistent Configuration" button on the alert post.
10. Now skyline health is showing healthy.

Re-initiate the OVF deployment. The OVF should now deploy successfully to the vSAN datastore without any errors.