"Datastores not available" while trying to update primary datastore via SDDC Manager in VCF 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Unable to update the primary datastore via the SDDC Manager user interface in a VMware Cloud Foundation (VCF) 9.1 environment. The environment contains a mix of Fibre Channel (FC) and non-FC (local) datastore.
• During the execution of a datastore query from the SDDC Manager GUI, the operation fails with the error " Datastores not available".
  
  

• /var/log/vmware/vcf/domainmanager/domainmanger.log

         DEBUG [vcf_dm,0000000000000000,0000] [c.v.e.s.c.c.v.s.FibreChannelStorageUtil,dm-exec-335]  Is Datastore <datastore_name> is a fc storage type false
         [INFO  [vcf_dm,0000000000000000,0000] [c.v.e.s.c.s.ClusterDatastoreUtil,dm-exec-335]  Given datastore Stub: moRef = (ManagedObjectReference: type = Datastore, value = datastore-##, serverGuid = null), binding = https://<vcenterfqdn>:443/sdk is not fibre channel datastore.

         ERROR [vcf_dm][c.v.e.s.c.s.DatastoreQueryHelper,http-nio-127.0.0.1-72##-exec-2]  Error while retrieving response object:
         java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException

• While checking the vCenter mob through https://<vcenter_server_IP/fqdn>/mob/?moid=domain-c#, the first datastore in the list is a local datatore.
  
       

Environment

VMware Cloud Foundation 9.1

Cause

When the backend API queries vCenter Server for mounted datastores, vCenter returns an arrayed list of datastore objects. If the first element in the returned array is a non-FC(local) datastore, a validation glitch in the SDDC Manager code fails to parse the object type correctly, causing the  update workflow to fail. The issue is intermittent and depends entirely on the object ordering returned by the vCenter Server API.

Resolution

• This is a known issue and will be fixed in VCF 9.1.1 patch release.
• To workaround the issue, the primary datastore allocation must be updated directly within the SDDC Manager database. Please contact Broadcom Support for assistance.