iSCSI Datastore creation fails despite successful vmkping with MTU 9000

Products

• VMware vSphere ESXi

Issue/Introduction

• Attempts to create or mount an iSCSI datastore fail, or existing iSCSI connections cycle between OFFLINE and ONLINE states.
• In /var/run/log/vmkernel.log, file of the ESXi host, you observe entries similar to:  

  iscsi_vmk: iscsivmk_StopConnection:736: vmhba64:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE"
  
  NMP device "naa.###" state in doubt; requested fast path state update
  
  iscsi_vmk: iscsivmk_ConnReceiveAtomic:478: vmhba64:CH:2 T:1 CN:0: Failed to receive data: Connection reset by peer
  
  iscsi_vmk: iscsivmk_TaskMgmtIssue:637: vmhba64:CH:2 T:1 L:0 : Task mgmt. "Abort Task" with itt=0xdc (refITT=0xd4) timed out.
  
  Partition: 607: Read of GPT entries (hdrlba=209715199) failed on "naa.###": I/O error
  
  Power-on Reset occurred on naa.###

 

Environment

VMware vCenter Server 

Cause

• This issue is typically caused by an MTU mismatch in the network path (Physical Switch, vSwitch, or Storage Array).
• While vmkping may have appeared successful, it often results in a false positive if the incorrect syntax was used. Specifically, if the vmkping was directed at the local VMkernel IP instead of the remote Storage Target IP, the packet never leaves the ESXi TCP/IP stack, bypassing the physical network and any MTU bottlenecks.
• Incorrect syntax may have been used with vmkping command.  Instead of iSCSI storage target IP, vmk adapter IP was used which will provide false positive results of vmkping succeeding.  Valid syntax to test MTU 9000 is: 

  vmkping -I vmk <iscsi target ip>  -d -s 8972

 

 

Resolution

• To resolve this, verify the MTU settings across the entire path and perform a valid jumbo frame ping test.

1. Display the virtual switch MTU settings 

   esxcfg-vswitch -l

2. Display the vmk MTU settings 

   esxcfg-vmknic -l

3. Display the vmnic MTU settings 

   esxcli network nic list

4. Get valid iSCSI storage target IPs.  

   esxcli iscsi adapter target portal list
   
   esxcli iscsi adapter discovery sendtarget list
   
   esxcli iscsi adapter discovery statictarget list

5. Valid vmkping command to test MTU 9000 from source ESXi host to target storage is:  

   vmkping -I <iSCSI vmk from step 1>  <iSCSI target IP from step 2>  -d -s 8972

 

 

 

Additional Information

Troubleshooting ESXi connectivity to iSCSI arrays using software initiators

https://knowledge.broadcom.com/external/article/311055/troubleshooting-esxi-connectivity-to-is.html

 

Testing VMkernel network connectivity with the vmkping command

https://knowledge.broadcom.com/external/article/344313/testing-vmkernel-network-connectivity-wi.html