FCoE Storage Connectivity Loss During VCF 9.1 deployment during the Cluster Configuration phase.

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

During the deployment of VCF 9.1.0.0400 (and other 9.1 releases), storage connectivity is lost shortly after the configuration of Distributed Switches (VDS). This results in an All Paths Down (APD) state, causing the vCenter Server and SDDC Manager to become inaccessible, ultimately failing the cluster configuration phase.

Environment

VMware Cloud Foundation 9.1.0.0
Distributed Virtual Switches (VDS) using FCoE (Fibre Channel over Ethernet) adapters.
QLogic/Marvell FCoE adapters (ql_fcoe/qedf drivers).

Cause

The issue is caused by the default fabric configuration of the Distributed Virtual Switches in VCF 9.1. When the VDS is brought up and pulls in the ESXi hosts, the default Link Layer Discovery Protocol (LLDP) settings interfere with FCoE FIP (FCoE Initialization Protocol) discovery. This leads to exhausted fabric retries and a subsequent loss of all device paths.

Resolution

Since the vCenter UI becomes inaccessible during the storage drop, the configuration must be corrected at the host level via CLI to restore connectivity, followed by a permanent adjustment in the vCenter UI.

Prerequisites
1.  Ensure you have SSH access to all ESXi hosts in the management cluster.
2.  Identify the specific names of the Distributed Virtual Switches created during deployment (e.g., <vds_name>).

Steps
1.  Monitor Cluster Configuration: During the VCF deployment, wait for the cluster configuration phase where the Distributed Virtual Switches appear on the hosts.
2.  Apply Host-Level Workaround: The VDS appears on each host, log in to each ESXi host via SSH and run the following commands for each uplink and VDS to restore connectivity:
    

    net-dvs -B 0x10001 -p uplink1 <vds_name_1>
    net-dvs -B 0x10001 -p uplink1 <vds_name_2>
    net-dvs -B 0x10001 -p uplink1 <vds_name_3>
    net-dvs -B 0x10001 -p uplink2 <vds_name_1>
    net-dvs -B 0x10001 -p uplink2 <vds_name_2>
    net-dvs -B 0x10001 -p uplink2 <vds_name_3>

Note: The number of VDS that are used are dependent on the number configured for your deployment - there may only be 1 VDS that needs to be modified, and the uplink# may vary based off your environment configuration.
    
3.  Verify Connectivity: Once the commands are executed on all hosts, storage paths should recover, and the vCenter Server/SDDC Manager should be accessible.
4.  Permanent UI Configuration: Log in to the vCenter Server UI to prevent recurrence:
    *   Navigate to Networking.
    *   Select the Distributed Switch.
    *   Go to Configure > Settings > Properties.
    *   Click Edit.
    *   Under Advanced, locate the Link Layer Discovery Protocol (LLDP) settings.
    *   Change the Operation mode from Both to Listen.
    *   Click OK.
    *   Repeat these steps for all Distributed Switches associated with the cluster.