Deploying a cluster with FCoE Storage, fails in SDDC manager 5.X, with the following error "Checking the Datastore availability"

Products

VMware SDDC Manager / VCF Installer VMware vCenter Server

Issue/Introduction

When deploying the aforementioned Cluster with FCoE storage, the task will fail, with the error "Checking the Datastore availability".

/var/run/log/vmkernel.log snippet shows an attempt to set that explicit VLAN ID...
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe:vmhba4:CreateFabric:112:Info: Fabric created: 0x431f7a571bb0
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe_spin_lock_init: lock = RportEvListLock
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe:vmhba4:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431f7a571bb0
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu64:2098487)ql_fcoe:vmhba3:SendFCoEVlanSolicitation:1477:Info: Sending FCoEVlanSolicitation request (0x3)
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe:vmhba4:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431f7a571bb0
YYYY-MM-DDT HH:MM:SSZ Wa(180) vmkwarning: cpu64:2098487)WARNING: ql_fcoe:vmhba3:FipVlanTimeoutWork:254: FIP VLAN Max Retries reached, cur vlan and pri: <VLAN_TBD>
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu64:2098487)ql_fcoe_spin_lock_init: lock = SessionListLock
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu64:2098487)ql_fcoe:vmhba3:CreateFabric:112:Info: Fabric created: 0x431f7a59d830
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu64:2098487)ql_fcoe_spin_lock_init: lock = RportEvListLock
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu64:2098487)ql_fcoe:vmhba3:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431f7a59d830
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe:vmhba4:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431f7a571bb0
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu64:2098487)ql_fcoe:vmhba3:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431f7a59d830

However it Fails below, after reaching the maximum number of attempts
YYYY-MM-DDT HH:MM:SSZ Wa(180) vmkwarning: cpu0:2098488)WARNING: ql_fcoe:vmhba4:FipDiscoveryTimeoutWork:125: Max retry exhausted for Fabric = 0x431f7a571bb0 vlan_id = 0x3ea
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe:vmhba4:StartPortLogout:1236:Info: Sess 0x431f7a58c740 port_id fffffe
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu0:2098488)ql_fcoe:vmhba4:CancelExchangeHandling:196:Info: Enter for Sess = 0x431f7a58c740
YYYY-MM-DDT HH:MM:SSZ In(182) vmkernel: cpu2:2098489)ql_fcoe:vmhba4:DeleteFabric:123:Info: Fabric 0x431f7a571bb0 000000 destroyed

While the Cluster is created, the task itself, cannot complete as the Datastore is inaccessible from a Host/ vCenter perspective.

Environment

SDDC Manager 5.x
VMware vCenter Server 8.x

Cause

As per Broadcom engineering, the ESX storage stack, doesn't impact the vlan configuration for the FCoE solution.
That should either be configured on the switch side, or the FCoE Firmware side, which is owned by the Vendor.

The default deployment, sets the value of LLDP to Both [from Available options = Listen, Advertise, Both] which seems to be the cause of the Failure.

Resolution

A fix for this issue is included in SDDC Manager version 5.2.4. However, if patching to this release is not immediately possible, the following workaround can be implemented to mitigate the issue.

To successfully complete the SDDC Manager task from the vSphere interface, the following sequence must be executed:

WORKAROUND

In order to complete the SDDC task from vSphere side, deploy a cluster with FCoE
A task failure is expected during the "Checking the datastore availability" phase. The task details will display the following progress message:
Description Checking the datastore availability
Progress Messages Datastore <Datastore_name> of datacenter datacenter_<TBD> is inaccessible
Following the sub-task failure, once the hosts are successfully deployed and joined to the cluster, the Link Layer Discovery Protocol (LLDP) setting on the Distributed Virtual Switch (DVS) must be modified to LISTEN.
vCenter > Networking > DVS > Configure > EDIT > Advanced > Discovery Protocol > Link Layer Discovery Protocol > select LISTEN
The affected ESXi hosts must be rebooted. Upon initialization, the Host Bus Adapters (HBAs) will be brought online.
Upon completion of the host reboots, the cluster deployment task must be retried from the SDDC Manager interface to finalize the deployment.

Additional Information

An additional, proactive workaround can be utilized from the SDDC Manager appliance to set the default LLDP OperationType prior to VDS creation during addCluster or addVI workflows:

An SSH session must be established to the SDDC Manager appliance, and privileges must be elevated to the root user.
The desired LLDP property must be appended to the Domain Manager application properties file by executing the following command: echo vcf.vds.lldp.operationtype="listen" >> /etc/vmware/vcf/domainmanager/application.properties Note: The accepted values for this property are 'both', 'listen', or 'advertise'.
The Domain Manager service must be restarted to apply the changes: systemctl restart domainmanager

Caution: The property value is case-sensitive. The exact lowercase value must be used (i.e., 'listen', not 'LISTEN'). If an incorrect value is set, the Domain Manager service will fail to restart. If a restart failure occurs, the correct value must be set, or the newly added entry must be completely removed from /etc/vmware/vcf/domainmanager/application.properties, and the Domain Manager service must be restarted again.