Datastore creation on a iscsi storage fails with error "Cannot change the host configuration. Failed to update disk partitions for..."

Products

• VMware vCenter Server
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

• Creating a new datastore on an iSCSI LUN fails with error:
  "Cannot change the host configuration. Failed to update disk partitions for /vmfs/devices/disks/naa.##############################."
  
  
  • Same error is seen when trying to create a datastore from the host client. 
    
    
    
• The error "Operation failed! An error occurred during host configuration: Unable to Find Head Extent Device Path" may also occur.
           

Environment

VMware vSphere ESXi 8.x

Cause

ISCSI network misconfiguration, such as MTU mismatch between the ESXi hosts, physical switches and the storage array is the cause for the failure.

While the initial TCP connection handshake and connection data is a small amount of data allowing the connection to be established, data operations sending large amounts of data will use the larger packet size. As the endpoints are configured with the larger MTU the packets leaving will be larger than 1500 which is configured at a point in the network path. When the packet reaches this point it is discarded.

The discarding of the large packet being discarded prevents the data from being received at the destination the operation is not completed or acknowledged and a BUS_BUSY SCSi code is returned. This is followed by an ABORT command being issued. You can them observe a message for "state in doubt, requesting fastpath update" before the connection is marked as OFFLINE. The connection is then reestablished and marked as ONLINE and the process repeats.

 

Cause Validation:

Review of /var/run/log/vmkernel.log file on the ESXi host reveals I/O aborts and iscsi connections changing status from online to offline and vice versa

####-##-##T##:##:##.###Z In(182) vmkernel: cpu111:2135720)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x2a (0x45daaff30f00, 97428766) to dev "naa.600######################" on path "vmhba64:C#:T#:L0" Failed:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu111:2135720)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430b57375a80 CmdSN 0x3
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu111:2135720)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600######################" state in doubt; requested fast path state update...

 

####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StartConnection:918: vmhba64:CH:4 T:1 CN:0: iSCSI connection is being marked "ONLINE"
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Sess [ISID: 00023d000005 TARGET: iqn.1988-1############## TPGT: 5 TSIH: 0]
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StartConnection:922: Conn [CID: 0 L: 10.#.#.#:39030 R: 10.#.#.#:3260]
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu170:2135964)WARNING: iscsi_vmk: iscsivmk_StopConnection:736: vmhba64:CH:4 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)

 

 

In addition to the above, vmkping to the iscsi target using jumbo frames fails with 100% packet loss.

vmkping -I vmk2 -s 8972 10.#.#.# -d
PING 10.#.#.# (10.#.#.#): 8972 data bytes

--- 10.#.#.# ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

However, connection is successful with standard MTU 1500

vmkping -I vmk2 -s 1472 10.#.#.#
PING 10.#.#.# (10.#.#.#): 1472 data bytes
1480 bytes from 10.#.#.#: icmp_seq=0 ttl=64 time=0.143 ms
1480 bytes from 10.#.#.#: icmp_seq=1 ttl=64 time=0.111 ms
1480 bytes from 10.#.#.#: icmp_seq=2 ttl=64 time=0.093 ms

--- 10.#.#.# ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.093/0.116/0.143 ms

Resolution

If jumbo frames are used for iSCSI connectivity, ensure the ESXi hosts, switches and storage array all have jumbo frames (MTU 9000) set. Refer KB318937

To test the network connectivity between ESXi and storage array with MTU 9000 refer KB311055