Lost access to Jumbo Frames enabled datastore post MTU change.

Products

VMware vSphere ESXi

Issue/Introduction

Symptom:

Datastores are either not able to show any files on the vCenter or are in inaccessible state.
Raw Device Mappings (RDMs) stored on the SAN are inaccessible.
The attached storage devices are shown under devices in the vSphere Client.
The attached storage devices are shown when you run the "esxcfg-scsidevs -c" command.
When you attempt to display the hexadecimal view of an affected storage device using the "hexdump" command, you will see a black screen, as you are unable to read the LUN.

Validation:

In the "var/run/log/vmkernel.log" you will see error logs similar to:

<Date>T09:16:34.475Z cpu4:33603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################4de" state in doubt; requested fast path state update...
<Date>T09:16:34.576Z cpu4:41898)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.################################4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:35.403Z cpu4:32809)ScsiDeviceIO
<Date>T09:17:56.853Z cpu14:33601)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.###.###.118:62638 R: 192.###.###.104:3260]
<Date>T09:17:56.953Z cpu2:32800)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x41368507dc00, 34012) to dev "naa.################################4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:17:56.957Z cpu9:69616)VMW_SATP_LOCAL: satp_local_updatePathStates:458: Failed to update path "vmhba37:<UUID>" state. Status=Transient storage condition, suggest retry

Environment

VMware vSphere ESXi

Cause

This issue is caused due to mismatched MTU in the iSCSI configuration.

Resolution

To resolve the issue, correct the MTU mismatch.

Determine the configured MTU by design, refer to the original iSCSI configuration documentation for your environment. Once attained you can then start applying that MTU size across your environment.
Ensure the allowed MTU is consistent on:

- The SAN array
- All physical network switches
- All virtual network switches
- All portgroups

To check the connectivity through CLI :

vmkping without any MTU parameter succeeds -

[root@esxi01 :~ ] vmkping -I vmk1 <Target storage IP>
PING <Target storage IP> : 56 data bytes
64 bytes from <Target storage IP>: icmp seq=0 ttl=64 time=0.260 ms
64 bytes from <Target storage IP>: icmp seq=1 ttl=64 time=0.222 ms
64 bytes from <Target storage IP>: icmp seq=2 ttl=64 time=0.285 ms

<Target storage IP> ping statistics
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.222/0.256/0.285 ms

vmkping with MTU 9000 fails -

[root@esxi01 :~ ] vmkping -I vmk1 <Target storage IP> -d -s 8972
PING <Target storage IP>: 8972 data bytes

<Target storage IP> ping statistics
3 packets transmitted, 0 packets received, 100% packet loss

Note: Once the MTU mismatch has been corrected, the storage should be accessible. ESXi/ESX hosts connected to the inaccessible storage may require a reboot to recover from the storage loss.