Rescanning takes a long time when using multiple VMkernel ports with port binding to access two or more storage arrays on different broadcast domains

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When using multiple VMkernel ports with port binding to access two or more storage arrays on different broadcast domains, you experience these symptoms:

• Rescan times take longer than usual.
• Incorrect number of paths are seen per device.
• In the syslog.log (ESXi 5.x) or vmkiscsid.log (ESX 4.x) file, you see entries similar to:
  
  T15:51:19Z iscsid: Login Target: iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 if=iscsi_vmk@vmk5 addr=10.1.38.30:3260 (TPGT:1 ISID:0x2)
  2012-03-05T15:51:19Z iscsid: Notice: Assigned (H34 T4 C1 session=c6, target=5/5)
  2012-03-05T15:51:19Z iscsid: Notice: Setting NODELACK for target=iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 (host=10.1.38.30)
  2012-03-05T15:51:19Z iscsid: cannot make a connection to 10.1.38.30:3260 (101,Network is unreachable)
  2012-03-05T15:51:19Z iscsid: Notice: Reclaimed Channel (H34 T4 C1 oid=5)
  2012-03-05T15:51:19Z iscsid: session login failed with error 4,retryCount=0
  2012-03-05T15:51:19Z iscsid: Login Target Failed: iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 if=iscsi_vmk@vmk5 addr=10.1.38.30:3260 (TPGT:1 ISID:0x2) err=4
  2012-03-05T15:51:19Z iscsid: Login Failed: iqn.2001-05.com.equallogic:0-8a0906-de0a06509-9620000002a4f4fe-z3-c1-t3-cust-01 if=iscsi_vmk@vmk5 addr=10.1.38.30:3260 (TPGT:1 ISID:0x2) Reason: 00040000 (Initiator Connection Failure)
  
  Note: For more information on log locations, see:
  
  
  • Location of ESXi 5.1 and 5.5 log files (2032076)
  • Location of ESXi 5.0 log files (2004201)
  • Location of ESX 3.0-4.1 log files (1021800)

Environment

VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.5
VMware ESXi 4.1.x Installable
VMware ESX 4.1.x

Cause

This issue occurs if you have different broadcast domains in conjunction with port binding, which is used to enable multipathing with a single target portgroup array.

Consider this sample iSCSI configuration, assuming the standard class-C netmask (255.255.255.0):

ESXi/ESX host:

• vmk1 IP – 10.10.37.1
• vmk2 IP – 10.10.38.1
• vmk3 IP – 10.10.37.2
• vmk4 IP – 10.10.38.2

Storage arrays:

• SAN A IP – 10.10.37.30
• SAN B IP – 10.10.38.30

When port binding is used, the software iSCSI stack asks all VMkernel ports to log in to all available targets on the storage arrays. This results in vmkernel ports vmk1 (10.10.37.1) and vmk3 (10.10.37.2) attempting to establish communication with the SAN B IP address (10.10.38.30), and vice versa.

Port binding requires that all target ports of the storage array must reside on the same broadcast domain as the vmkernel ports because routing is not supported with port binding.

Resolution

To work around this issue, use one of these options:

• Ensure you have both arrays on the same broadcast domain, resulting in all vmkernel ports being able to access all target port groups.
• Split the SANs into Software/Hardware Independent iSCSI, Software Hardware Dependent iSCSI, or two Hardware iSCSI domains.

Note: For some storage vendors that utilize ICMP pings to determine the link status between the array and the ESXi/ESX host, try disabling the ping requests to the unreachable vmkernel ports.

Additional Information

Location of ESX 3.0-4.1 log files
Location of ESXi 5.0 log files
Location of ESXi 5.1 and 5.5 log files
Considerations for using software iSCSI port binding in ESX/ESXi