Lost access to Jumbo Frames enabled datastore post MTU change.

Products

VMware vSphere ESXi

Issue/Introduction

When running VMware ESXi/ESX, a host with an attached iSCSI SAN array configured to use Maximum Transmission Unit (MTU) 9000 (jumbo frames), experiences symptoms similar to:

Configured storage devices and related datastores are inaccessible
Raw Device Mappings (RDMs) stored on the SAN are inaccessible
The attached storage devices are shown under devices in the vSphere Client
The attached storage devices are shown when you run the esxcfg-scsidevs -c command
When you attempt to display the hexadecimal view of an affected storage device using the hexdump command, you will see a black screen, as you are unable to read the LUN.
You see error messages in the vmkernel logs similar to:

<Date>T09:16:34.475Z cpu4:33603)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" state in doubt; requested fast path state update...
<Date>T09:16:34.576Z cpu4:41898)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:34.718Z cpu4:41898)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:34.787Z cpu8:66250)VMW_SATP_LOCAL: satp_local_updatePathStates:458: Failed to update path "vmhba37:<UUID>" state. Status=Transient storage condition, suggest retry
<Date>T09:16:34.862Z cpu4:33888)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:35.003Z cpu4:33603)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:35.104Z cpu4:33616)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" state in doubt; requested fast path state update...
<Date>T09:16:35.147Z cpu4:33616)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:35.288Z cpu4:33603)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x413685059c40, 34048) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:16:35.403Z cpu4:32809)ScsiDeviceIO

<Date>T09:17:56.853Z cpu14:33601)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.131.118:62638 R: 192.168.131.104:3260]
<Date>T09:17:56.953Z cpu2:32800)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x41368507dc00, 34012) to dev "naa.600XXXXXXXXXXXXXXXXXXXXXXXXX4de" on path "vmhba37:<UUID>" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
<Date>T09:17:56.957Z cpu9:69616)VMW_SATP_LOCAL: satp_local_updatePathStates:458: Failed to update path "vmhba37:<UUID>" state. Status=Transient storage condition, suggest retry

Environment

VMware vSphere ESXi

Cause

This issue is caused by a change in MTU in the iSCSI storage configuration.

Resolution

To resolve the issue, correct the MTU mismatch.

Determine the configured MTU by design, refer to the original iSCSI configuration documentation for your environment. Once attained you can then start applying that MTU size across your environment.
Ensure the allowed MTU is consistent on:

The SAN array
All physical network switches
All virtual network switches
All portgroups

Note: Once the MTU mismatch has been corrected, the storage should be accessible. ESXi/ESX hosts connected to the inaccessible storage may require a reboot to recover from the storage loss.