EqualLogic SAN over DCB Solution for VMware ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information about Partner Verified and Supported Products (PVSP) for the Internet SCSI (iSCSI) over Data Center Bridging (DCB) Solution for ESX from partner Dell, Inc. The iSCSI over DCB solution for ESX consists of Dell™ EqualLogic™ PS Series storage arrays supporting 10GbE in conjunction with host adapters and switch components that provide Enterprise iSCSI over DCB functionality.

Note: The Partner Verified and Supported Products (PVSP) policy implies that the solution is not directly supported by VMware. For issues with this configuration, contact Dell directly. See the Support Workflow on how partners can engage with VMware. It is Dell's responsibility to verify that the configuration functions with future vSphere major and minor releases, as VMware does not guarantee that compatibility with future releases is maintained.

Disclaimer: The partner products referenced in this article are hardware devices that are developed and supported by stated partners. Use of these products are also governed by the end user license agreements of the partners. You must obtain the application, support, and licensing for using these products from the partners. For more information, see Support Information in this article.

Resolution

Solution Overview

The supportable solution for providing Enterprise iSCSI over DCB converged network within the VMware environment includes:

• Converged network adapters (CNAs) that have the ability to manage all of the DCB functionality within its hardware engine without any assistance from ESX. See the EqualLogic Compatibility Matrix for specific vendors and models.
• Switches that support these DCB standard features:
  • Per Priority Flow Control (PFC)
  • Enhanced Transmission Selection (ETS)
  • Data Center Bridging Exchange (DCBx)
  • iSCSI TLV (Enterprise iSCSI over DCB)
    
    Note: See the EqualLogic Compatibility Matrix for specific vendors and models.
     
• EqualLogic PS-Series 10Gb Ethernet arrays Array Firmware 5.1.x or later

This image shows a diagram of just such a device. As the figure shows, there are two different devices presented to the ESXi kernel via the CNA driver. Each of these devices is actually managed independently by the CNA hardware and each device is configured appropriately for the required DCB settings needed to permit shared usage of the converged network infrastructure.

 

 

Solution Details

Host Configuration

CNA Functionality Requirements

CNAs operate in one of two modes based on their dependency on the host OS (VMware) for some of its functionality. A CNA can operate in:

• Simple NIC mode, where all of the functionality for the TCP/IP and iSCSI functions are provided for by the host OS
• Hardware offload mode for TCP and/or iSCSI, where there is no management of these functions by the host OS, but is completely managed by the CNA hardware device.

The first method requires specific kernel based code to manage the different types of traffic (LAN/Client, SAN, Cluster, Management) and based on the source or destination information, modify the Ethernet frame with the proper tags to allow the host converged network adapters and switches to properly manage the frame from the host to the eventual destination. Since this code does not currently exist in the VMware kernel for iSCSI, this method is not possible with current VMware versions.

The second method requires that the CNA manage all of the traffic in proxy to the host operating system. This means that the host operating system is not required to have any knowledge of the DCB environment for it to be able to participate in the DCB converged network. To do this, the CNA must have a way of properly routing the traffic between the network and the host operating system. In this method, that would require that the CNA provide multiple personalities to the host operating system via its in-box driver. These personalities are represented by pseudo-devices. A CNA with this functionality has the ability to present to the OS a standard Ethernet NIC pseudo-device and a separate SCSI Storage Controller pseudo-device. Additionally, the CNA may offer some form of network partitioning that allows the Ethernet NIC device to be further partitioned into multiple virtual kernel NIC ports.

To be able to function in this second mode of operation, the CNA must have the ability to operate in either Host Bus Adapter mode or iSCSI Hardware Offload mode. Today, there are several CNAs that meet this criteria. For details on how to determine which CNAs have been validated by Dell’s Interoperability Test Lab for use within this solution, see the EqualLogic Compatibility Matrix.

CNA Configuration

For iSCSI to be properly managed by the DCB infrastructure, traffic must be able to be sorted and categorized based on pre-defined class of service settings being used within the DCB domain. There are two methods for this to be accomplished:

• The host CNAs can be manually configured for these settings
  
  Or
   
• The host CNAs can accept the DCB settings from the switch(es) to which the CNA is connected.

By industry ad-hoc agreement, it is customary for DCB settings to be maintained within the switching domain and that all edge devices (such as CNAs and arrays) be in DCB Willing mode. In this mode, the edge devices receive their DCB settings from the switch upon initial port link up. Any CNA used within this solution must be configured in DCB willing mode. Since the VMware environment is unaware of the DCB settings, it may be necessary that all iSCSI over DCB configuration of the selected CNAs be done via the CNA’s BIOS configuration interface available during server Power-On Self Test process or other pre-boot management processes like the Dell Lifecycle Manager. Depending on CNA vendor implementations, it may be possible to use the vendor’s vCenter Server plug-in module to configure the appropriate DCB settings.

Each CNA vendor has unique set of steps to properly configure the CNA into DCB willing mode and to enable support for iSCSI over DCB. See the vendor’s product documentation for the specific steps. In general, these steps include:

1. Enable iSCSI Offload or iSCSI HBA mode.
2. Select DCB Willing or Accept DCB Settings from Switch (or similar setting) to enable DCB willing mode.
3. Install appropriate iSCSI Offload/HBA VMware drivers provided by the vendor and located on the VMware Hardware Compatibility List. See Support Information to find out how to identify the supported network adapters and the tested driver versions.

Switches

Switch Functionality Requirements

Many switch vendors claim support for DCB, but the majority of those that do claim DCB support only provide support for FCoE. It is very important that the switch vendor provide a full suite of DCB standard functionality to ensure that the switch will provide the functionality of the EqualLogic PS arrays within the DCB domain.

 

The specific features that are required for support of iSCSI over DCB with EqualLogic are:

• PFC – In this solution, iSCSI traffic must not be blocked by any other types (streams) of network traffic such as the LAN or other network use cases. Per Priority Flow Control prevents blocking of resources from other classes of service within the DCB domain. Any switch for this solution must allow the iSCSI traffic to be assigned to a unique class of service from all other traffic types.
   
• ETS – In this solution, iSCSI SAN traffic must be guaranteed a minimum amount of total bandwidth throughout the DCB infrastructure domain. Enhanced Transmission Selection is the mechanism that defines and enforces the bandwidth allocation as a percentage of the link bandwidth.
   
• DCBx with iSCSI TLV – The switch must be able to communicate to other DCB capable devices the unique iSCSI settings within the DCB domain. This is done by transmitting an iSCSI TLV Ethernet frame within the DCB Capabilities Exchange process.

Dell tests a variety of switches from multiple vendors as part of its regular process of array development. If a switch provides the required DCB functionality, Dell tests the switch for DCB compliance for iSCSI over DCB. If a switch has been determined to support these requirements, Dell indicates that support within its EqualLogic Compatibility Matrix.

 

This image provides an example of that designation within the EqualLogic Compatibility Matrix:

Note: This example indicates that that iSCSI over DCB suitable for EqualLogic solutions is available for the Force10 s4810 product when using switch firmware 8.3.12.x, but is not supported that same switch if firmware is 8.3.7.2.

 

 

 

Switch Configuration

Each switch has a unique configuration process to enable iSCSI over DCB. In general, the steps include:

1. Enable DCB functionality.
2. Identify iSCSI or TCP Port 3270 for a unique class of service (default generally accepted as CoS:4).
3. Assign iSCSI class of service to a DCB Priority Group.
4. If required, define the iSCSI assigned Priority Group as a no-drop priority group.
5. Define the ETS bandwidth allocation between all active DCB priority groups.

Notes:

• ETS is configured based on percentage of link’s operating bandwidth. For example, an ETS setting of 30% would indicate 3Gb of bandwidth for a 10Gb link. It would also indicate 12Gb of bandwidth on a 40Gb link.
• ETS settings are a guaranteed minimum bandwidth allocation for the priority group. If additional bandwidth is available at any given time, a priority group may use additional bandwidth until it is required. Also, if classes assigned to the priority group are not transmitting, the bandwidth can be made available to other priority groups until such time as the priority group bandwidth is again required.
• Dell recommends isolating iSCSI traffic into its own VLAN to provide an additional layer of security.

For each switch that Dell indicates as DCB Support in the EqualLogic Compatibility Matrix, a switch reference architecture is provided that describes the specific steps for configuring DCB for that switch model. While the specific settings for Class of Service and Priority Group settings may not be those required for the network, replace the documented values with the required values for the environment.

Dell EqualLogic PS Series Storage Arrays

All PS Series 10Gb Ethernet storage arrays are DCB ready and can have DCB capabilities enabled when using Array Software version 5.1 or later. When DCB is enabled on a Storage Group, the arrays are placed into a DCB Willing mode whereby they will accept DCB domain settings from the DCB configured switch to which they are attached. This is the only method the arrays can use to be configured with the appropriate DCB settings for PFC, Class of Service, ETS, Priority Groups, etc. that make up the definition of the DCB domain.

To configure DCB on the EqualLogic SAN Group:

1. Ensure that Array Software 5.1 or later is installed on all PS arrays in the Storage Group 2. Ensure that DCB is enabled on the PS Array Group:
   1. Click Group Manager > Group Configuration > Advanced tab > Network Management.
   2. Select Enable DCB:
      
      
      

   3. Enter the VLAN ID that is being used for iSCSI or Converged network traffic on the switch infrastructure.
      
      Note: The default VLAN ID is listed as VLAN 0, but that VLAN should not be used in production environments. iSCSI traffic should be isolated to a separate, non-default VLAN from other VLANs.

2. Verify DCB settings

   1. Select a Group Member array.

   2. Click the Network tab.

   3. Right-click on any array member Ethernet port and select DCB Details.

      Notes:

      • iSCSI should be listed in a Group and be listed as Lossless.
      • DCB VLAN ID should not be the default VLAN (typically VLAN0 or VLAN1 depending on switch vendor). Many switch vendors ignore or strip off the VLAN ID Tag within the Ethernet frame on the default VLAN.
      • VLAN ID must match the VLAN assigned to iSCSI SAN traffic within the switch infrastructure. If this VLAN ID is not properly set, DCB will not work and host-target connections may not be possible.

Dell Verification Details

Over the last three years Dell has attended numerous industry plugfests sponsored by the Ethernet Alliance’s Data Center Bridging Consortium, the University of New Hampshire (UNH) iSCSI and DCB consortium plugfests, and is listed in the UNH IOL Interoperability Integrators Matrix.

The UNH IOL iSCSI over DCB test matrix can be obtained here.

Dell Internal Testing Results:

 

Test	Description	Results
DCB Support-Configuration of DCBx	To determine the ability of the switch to be configured for Data Center Bridging and the ability of EQL arrays and CNAs to negotiate properly DCB settings with the switch.   Tests the ability to configure proper version of DCBx.	Pass
DCB Support-Configuration of ETS	Tests the ability to configure and the ability to provide required bandwidth for each traffic type, as well as identify iSCSI traffic via iSCSI TLV.	Pass
DCB Support-Configuration of PFC	Tests the ability to configure a lossless class of traffic.	Pass
DCB Support-Configuration of iSCSI TLV	Tests the ability to configure a lossless class of traffic assigned to an iSCSI traffic stream.	Pass
DCB Support-Proper Advertisement of iSCSI TLV	This test is to determine the ability of the CNA to properly receive DCBx configuration settings related to a dedicated iSCSI CoS.	Pass
DCB Support-Proper Management of converged traffic (2-Streams)	This test is to determine if the configured DCB functionality of a DCB capable switch is implemented in the manner expected through the use of sample workloads from multiple priorities and traffic classes.	Pass
DCB Baseline-Storage only	This test is to set a baseline for iSCSI only traffic in a DCB environment. The results of this test will be used as a comparison in the Management of Converged Traffic test to verify that ETS percentages are being managed correctly.	Pass
Reverse Burst-100% LAN then add storage	This test is to ensure that adding lossless traffic to an already utilized switch running lossy traffic does not cause undesired behavior.	Pass
DCB compliance during IO	Verification of proper transmit of iSCSI frames withing DCB environment including transmit of PFC frames, responding to PFCs, enforcing ETS assignments, and maintaining lossless behavior for the iSCSI queue.	Pass
DCB Steady State Behavior	This test is to determine if the configured ETS bandwidth of a DCB capable switch is implemented in the manner expected through the use of sample workloads from multiple priorities and traffic classes. It is expected that the switch will manage the traffic to the configured specification in a timely manner, without causing disruptions or errors in the traffic flows.	Pass

Support Information

The first line of support for this solution is Dell’s EqualLogic Customer Support (https://support.equallogic.com/). All Dell customers are required to have active support agreements for all components within the solution (Servers, OS, CNA, Switch, and PS Arrays).

A minimum of ProSupport Next Day Response (or vendor equivalent) is required for all components in the SAN/LAN converged solution defined by the knowledge base article and support will be provided based on the limits of that contractual agreement.

Candidate CNA and Switch components must meet this criteria:

• CNAs and Switches must be listed on the Dell EqualLogic Compatibility Matrix
• CNAs must indicate support for DCB for the VMware version being deployed (5.1 or later).
• Switches must indicate support for DCB with the deployed version of PS Series Array Software
• CNAs must be listed on the VMware Hardware Compatibility List

EqualLogic VMware Integration Software and Documentation can be obtained here. (Login and product registration required)

VMware Release To Release Support

Because DCB support is self-contained, resident in the CNA hardware, switch hardware and the iSCSI array, release to release compatibility with ESXi is expected.

Additional Resources

• VMware IO Compatibility Guide
• Dell EqualLogic Configuration Guide
• Dell Switch Configuration Guides
• Dell Rapid EqualLogic Configuration Portal
• Dell EqualLogic Whitepapers

VMware Release to Release compatibility

Dell’s continuing interoperability test program includes integration of future VMware releases. The Dell EqualLogic Compatibility Matrix will provide additional details about the suitability of future version of VMware ESXi.