Datastore creation on a iscsi storage fails with error
Article ID: 388049
Updated On:
Products
Issue/Introduction
Symptoms :
- Creating a new datastore on an iSCSI LUN fails with error:
"Cannot change the host configuration. Failed to update disk partitions for /vmfs/devices/disks/naa.##############################." - Same error is seen when trying to create a datastore from the host client.
- Same error is seen when trying to create a datastore from the host client.
- The error "
Operation failed! An error occurred during host configuration: Unable to Find Head Extent Device Path" may also be seen:
- In certain instances, disk partition may not be detected at all during running of "New Datastore" wizard:
Environment
VMware vSphere ESXi 8.x
Cause
- ISCSI network misconfiguration, such as MTU mismatch between the ESXi hosts, physical switches and the storage array is the cause for the failure.
- While the initial TCP connection handshake and connection data is a small amount of data allowing the connection to be established, data operations sending large amounts of data will use the larger packet size. As the endpoints are configured with the larger MTU the packets leaving will be larger than 1500 which is configured at a point in the network path. When the packet reaches this point it is discarded.
- The discarding of the large packet being discarded prevents the data from being received at the destination the operation is not completed or acknowledged and a
BUS_BUSYSCSi code is returned. This is followed by anABORTcommand being issued. You can them observe a message for "state in doubt, requesting fastpath update" before the connection is marked asOFFLINE. The connection is then reestablished and marked asONLINEand the process repeats.
Cause Validation:
- Review of /var/run/log/vmkernel.log file on the ESXi host reveals I/O aborts and iscsi connections changing status from online to offline and vice versa
####-##-##T##:##:##.###Z In(182) vmkernel: cpu111:2135720)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x2a (0x45daaff30f00, 97428766) to dev "naa.600######################" on path "vmhba64:C#:T#:L0" Failed:####-##-##T##:##:##.###Z In(182) vmkernel: cpu111:2135720)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430b57375a80 CmdSN 0x3####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu111:2135720)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600######################" state in doubt; requested fast path state update...
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StartConnection:918: vmhba64:CH:4 T:1 CN:0: iSCSI connection is being marked "ONLINE"####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Sess [ISID: 00023d000005 TARGET: iqn.1988-1############## TPGT: 5 TSIH: 0]####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StartConnection:922: Conn [CID: 0 L: 10.#.#.#:39030 R: 10.#.#.#:3260]####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StopConnection:736: vmhba64:CH:4 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
- In addition to the above, vmkping to the iscsi target using jumbo frames fails with 100% packet loss.
vmkping -I vmk2 -s 8972 10.#.#.# -dPING 10.#.#.# (10.#.#.#): 8972 data bytes
--- 10.#.#.# ping statistics ---3 packets transmitted, 0 packets received, 100% packet loss
However, connection is successful with standard MTU 1500
vmkping -I vmk2 -s 1472 10.#.#.#PING 10.#.#.# (10.#.#.#): 1472 data bytes1480 bytes from 10.#.#.#: icmp_seq=0 ttl=64 time=0.143 ms1480 bytes from 10.#.#.#: icmp_seq=1 ttl=64 time=0.111 ms1480 bytes from 10.#.#.#: icmp_seq=2 ttl=64 time=0.093 ms
--- 10.#.#.# ping statistics ---3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0.093/0.116/0.143 ms
- In some scenarios, vmkping results may be successful but syslog.log do report storage array fails to respond to ESXi requests and these requests times-out.
(from var/run/log/syslog.log)YYYY-MM-DDTHH:MM:SS.604Z Wa(28) iscsid[55608520]: connection 3:0 (iqn.####-##.com.####storage:flasharray.################ if=iscsi_vmk@vmk# addr=#.#.#.#:3260 (TPGT:2500 ISID:0x3) (T0 C2)) Nop-out timeout after 10 sec in state (3).
Resolution
Feedback
