Rescanning takes a long time when using multiple VMkernel ports with port binding to access two or more storage arrays on different broadcast domains

Products

• VMware vSphere ESX 4.x - View
• VMware vSphere ESX 5.x

Issue/Introduction

Symptoms:

When using multiple VMkernel ports with port binding to access two or more storage arrays on different broadcast domains, you experience these symptoms:

• Rescan times take longer than usual.
• Incorrect number of paths are seen per device.
• In the syslog.log (ESXi 5.x) or vmkiscsid.log (ESX 4.x) file, you see entries similar to:
  
  [YYYY-MM-DDTHH:MM:SS] iscsid: Login Target: iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 if=iscsi_vmk@vmk5 addr=10.1.38.30:3260 (TPGT:1 ISID:0x2)
  [YYYY-MM-DDTHH:MM:SS] iscsid: Notice: Assigned (H34 T4 C1 session=c6, target=5/5)
  [YYYY-MM-DDTHH:MM:SS] iscsid: Notice: Setting NODELACK for target=iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 (host=10.1.38.30)
  [YYYY-MM-DDTHH:MM:SS] iscsid: cannot make a connection to 10.1.38.30:3260 (101,Network is unreachable)
  [YYYY-MM-DDTHH:MM:SS] iscsid: Notice: Reclaimed Channel (H34 T4 C1 oid=5)
  [YYYY-MM-DDTHH:MM:SS] iscsid: session login failed with error 4,retryCount=0
  [YYYY-MM-DDTHH:MM:SS] iscsid: Login Target Failed: iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 if=iscsi_vmk@vmk5 addr=10.1.38.30:3260 (TPGT:1 ISID:0x2) err=4
  [YYYY-MM-DDTHH:MM:SS] iscsid: Login Failed: iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 if=iscsi_vmk@vmk5 addr=10.1.38.30:3260 (TPGT:1 ISID:0x2) Reason: 00040000 (Initiator Connection Failure)
  
  Note: For more information on log locations, see:
  
  
  • Location of ESXi 5.1 and 5.5 log files (2032076)
  • Location of ESXi 5.0 log files (2004201)

Cause

This issue occurs if you have different broadcast domains in conjunction with port binding, which is used to enable multipathing with a single target portgroup array.

Consider this sample iSCSI configuration, assuming the standard class-C netmask (255.255.255.0):

ESXi/ESX host:

• vmk1 IP – 10.10.37.1
• vmk2 IP – 10.10.38.1
• vmk3 IP – 10.10.37.2
• vmk4 IP – 10.10.38.2

Storage arrays:

• SAN A IP – 10.10.37.30
• SAN B IP – 10.10.38.30

When port binding is used, the software iSCSI stack asks all VMkernel ports to log in to all available targets on the storage arrays. This results in vmkernel ports vmk1 (10.10.37.1) and vmk3 (10.10.37.2) attempting to establish communication with the SAN B IP address (10.10.38.30), and vice versa.

Port binding requires that all target ports of the storage array must reside on the same broadcast domain as the vmkernel ports because routing is not supported with port binding.

Resolution

To work around this issue, use one of these options:

• Ensure you have both arrays on the same broadcast domain, resulting in all vmkernel ports being able to access all target port groups.
• Split the SANs into Software/Hardware Independent iSCSI, Software Hardware Dependent iSCSI, or two Hardware iSCSI domains.

Note: For some storage vendors that utilize ICMP pings to determine the link status between the array and the ESXi/ESX host, try disabling the ping requests to the unreachable vmkernel ports.

Additional Information

Location of ESXi 5.0 log files
Location of ESXi 5.1 and 5.5 log files
Considerations for using software iSCSI port binding in ESX/ESXi