iSCSI Datastores Missing After Upgrade to ESXi 9.1

Products

• VMware vSphere ESXi

Issue/Introduction

After upgrading the ESXi hosts from 9.0.2 to version 9.1 the LUNs are discovered and have an active path for ESXi. However, the datastores do not display any partitions, due to failing to read from them. 

 

Errors in: /var/log/vmkernel.log 

• Partition table read from device naa.device failed: I/O error
• Read from both primary and secondary gpt table failed on naa.device
• Read of GPT entries (hdrlba=85899345919) failed on "naa.device": I/O error

 

Missing partition tables from LUNs, where other hosts (that have not been upgraded), still show the datastore active and its partitions. 

1. esxcli storage core device partition list
   • If the LUNs/Devices for the datastores only have 'Partition 0' this means the device is seen, but the data partitions cannot be read. 

Environment

ESXi 9.1

Cause

In ESXi version 9.1, when jumbo frames is configured. The connection will be tested with a full frame of 9000 MTU, when initially mounting storage. 

An MTU mismatch (ie, vSwitch, VMK, or Switch port) causes a truncated frame where this could cause failure to read the partition tables from the LUNs. 

Resolution

Verify active port (VMNIC) on the ISCSI port group, and ensure the correct NIC and port group is in use for iSCSI. 

1. Log into ESXi host through SSH. [Using ESXi Shell in ESXi]
2. Enter command: esxtop
   • Press "n" and verify the VMnic# for the iSCSI VMK# is actively being used. (If not, this is due to traffic shaping on the DVS port group). 
3. Use KB: [Troubleshooting ESXi connectivity to iSCSI arrays using software initiators]
   • Specifically: vmkping -d -s 8972 <target_ip> ; to test with jumbo frames. 

 

If this fails to send a full frame for Jumbo frames, where the VMK is configured for 9000. Then confirm the correct VMNIC is in use for the iSCSI and also confirm the DVS switch itself. 

1: Enable Jumbo Frames on a vSphere Distributed Switch.

2: Confirm on Switch that the iSCSI VMNIC(s) are configured for Jumbo Frames. 

 

After configuration is corrected, rescan the storage on that ESXi host. 

1: Right click ESXi host within vCenter. 

2: Under Storage, click 'Rescan storage Devices', check both (HBA's and Device changes); and click "Okay". 

3: Confirm Datastores are not visible for that ESXi host. 

Additional Information

Enable Jumbo Frames on a vSphere Distributed Switch.

Testing iSCIS connectivity to storage array