Fibre channel based SDDC Manager cluster creation workflow failure with error "VSPHERE_DATASTORE_NOT_FOUND Datastore not found on the host".

Products

• VMware Cloud Foundation

Issue/Introduction

In order to have a successful add cluster/domain/host operations.

Symptoms:
FC datastore is not mounted on participating hosts

Some of these symptoms can occur (if not all):

Add cluster/domain/host workflow failed with below error

XXXX-XX-XXT16:39:11.295+0000 ERROR [vcf_dm,a3b2e1c3800d43d5,18bc] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-19] [P850PT] VSPHERE_DATASTORE_NOT_FOUND Datastore xxxxxx not found on the host xxxxxx
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Datastore xxxxxxxx not found on the host xxxxxxxxxxxxx
 at com.vmware.vcf.common.fsm.plugins.action.impl.ConfigureFibreChannelDatastoreAction.validateFibreChannelExistenceOnHost(ConfigureFibreChannelDatastoreAction.java:125)
 at com.vmware.vcf.common.fsm.plugins.action.impl.ConfigureFibreChannelDatastoreAction.preValidate(ConfigureFibreChannelDatastoreAction.java:57)
 at com.vmware.vcf.common.fsm.plugins.action.impl.ConfigureFibreChannelDatastoreAction.preValidate(ConfigureFibreChannelDatastoreAction.java:39)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:18)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
 at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
 at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:545)
 at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
 at jdk.internal.reflect.GeneratedMethodAccessor1369.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
 at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:88)
 at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:73)
 at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
 at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:829)

Environment

VMware Cloud Foundation 4.0.x
VMware Cloud foundation 5.x
VMware Cloud Foundation 3.0.x
VMware Cloud Foundation 4.5.x
VMware Cloud Foundation 9.x
VMware Cloud Foundation 9.1

 

Cause

If you are using VMFS on FC storage for the workload domain, you must create the datastore on the hosts and configure zoning and mounting of associated volumes.

Refer to : https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/working-with-workload-domains-admin/about-virtual-infrastructure-workload-domains-admin/prerequisites-for-a-workload-domain-admin.html  

-- 

VCF 9.x 

In some case the new datastore gets created with label name (1) on different hosts which cause failure for datastore detection. 

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:
Follow the below mentioned step:

1. Login to host and create a FC data store by going to storage > datastores > New Datastore
2. Select "Create New VMFS data store"
3. Give a name for data store and select one of the available FC LUN.
   • Note:  Use the same FC datastore name in the add cluster workflow that is  either in UI/spec
4. Once data store creation is done restart the failed workflow.

-- 

VCF 9.x

Workaround: 

Follow the below mentioned step: 

• Unmount Datastore from VC
• Rename datastore from VC UI
• Rescan storage on both Host and vCenter
• Renamed Datastore via CLI on Hosts

command: 
# vim-cmd hostsvc/datastore/rename "OldDatastoreName" "NewDatastoreName"