Software iSCSI connections drop and reconnect during I/O operations

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• iSCSI connections drop and reconnect during I/O operations
• Unable to create a VMFS volume on the presented LUN
• If an existing VMFS volume is presented, performing I/O writes to the LUN causes it to the drop the connection
• When trying to create a VMFS datastore on the presented iSCSI LUN, the vmkernel logs contain entries similar to:

Sep 23 07:22:02 Hostd: [2010-09-23 07:22:02.006 19A76B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.host.DatastoreSystem.createVmfsDatastore-63256
Sep 23 07:22:02 Hostd: [2010-09-23 07:22:02.040 19A76B90 verbose 'Partitionsvc'] InvokePartedUtil /sbin/partedUtil "set" "/vmfs/devices/disks/naa.6000eb3883fe658b000000000000005d" "1 128 20964824 251 0"
Sep 23 07:22:02 Hostd: [2010-09-23 07:22:02.660 19A76B90 info 'Hostsvc::DatastoreSystem'] Formatting vmfs partition Sep 23 07:22:02 Hostd: [2010-09-23 07:22:02.660 19A76B90 verbose 'FSVolumeProvider'] FormatVmfs called

Sep 23 07:22:17 iscsid: Nop-out timedout after 10 seconds on connection 1:0 state (3). Dropping session.
Sep 23 07:22:17 vmkernel: 13:01:38:03.830 cpu10:4768)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba33:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE"
Sep 23 07:22:17 vmkernel: 13:01:38:03.830 cpu10:4768)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.<domain>:smgsan01:93:test TPGT: 1 TSIH: 0]

Sep 23 07:22:20 vmkernel: 13:01:38:06.601 cpu10:4768)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba33:CH:0 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound Sep 23 07:22:20 vmkernel: 13:01:38:06.601 cpu10:4768)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.<domain>:smgsan01:93:test TPGT: 1 TSIH: 0] Sep 23 07:22:20 vmkernel: 13:01:38:06.601 cpu10:4768)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.5.106:62961 R: 192.168.5.20:3260]

Environment

VMware ESXi 3.5.x Embedded
VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Installable
VMware ESXi 4.0.x Installable
VMware ESX 4.0.x
VMware ESXi 4.0.x Embedded
VMware ESX Server 3.0.x
VMware ESX Server 3.5.x

Resolution

To resolve this issue, verify:

• The connectivity from ESX hosts to the iSCSI array
• The MTU settings for the vSwitch and VMNICs
• Check the duplex settings on the vSwitch, Physical Switch and iSCSI SAN and make sure they are same across the board

Note:

• This issue is seen to occur on Juniper switches. On some Juniper switches, the default MTU is 1514. This issue may occur if the MTU for the connected port is set to a value other than 1514. Changing the port MTU back to 1514 resolves this issue. For more information, see Configuring the Media MTU section of the JUNOS 8.4 Network Interfaces Configuration Guide or contact Juniper support.
  
  The preceding link was correct as of February 28, 2010. If you find the link is broken, provide feedback and a VMware employee will update the link.
   
• This issue also occurs when using the DELL switch Power connect 8024. You experience this issue when the MTU of the switch and the vSwitch on the ESX host are both set to 9000. To resolve this issue, the switch should be configured with a MTU more than the MTU configured on the ESX host.

Additional Information

For related information, see Troubleshooting iSCSI array connectivity (1003681).I/O 操作中にソフトウェア iSCSI 接続が切断および再接続される