vMotion of VMs residing on VVOL Datastore fails with error 'A general system error occurred'
Article ID: 394246
Updated On:
Products
Issue/Introduction
Attempt to create a new VM on the VVOL Datastore fails with message 'Cannot complete file creation operation'
Environment
ESXi
VVOL
Cause
This is due to a MTU mismatch.
While we are able to ping the target with defragmented packet size of 1472 (1500 - 28 header), defragmented packet of 8972 (9000 - 28 header) fails.[root@esxi##:~] vmkping -I vmk1 ##.##.##.## -d -s 8972PING ##.##.##.## (##.##.##.##): 8972 data bytes--- ##.##.##.## ping statistics ---3 packets transmitted, 0 packets received, 100% packet loss[root@esxi##:~] vmkping -I vmk1 ##.##.##.## -d -s 1472PING ##.##.##.## (##.##.##.##): 1472 data bytes1480 bytes from ##.##.##.##: icmp_seq=0 ttl=64 time=0.213 ms1480 bytes from ##.##.##.##: icmp_seq=1 ttl=64 time=0.185 ms1480 bytes from ##.##.##.##: icmp_seq=2 ttl=64 time=0.210 ms--- ##.##.##.## ping statistics ---3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0.185/0.203/0.213 ms
The following is logged in vmkwarning log####-##-##T##:##Z cpu24:2099084)WARNING: iscsi_vmk: iscsivmk_StartConnection:919: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"####-##-##T##:##Z cpu24:2099084)WARNING: iscsi_vmk: iscsivmk_StartConnection:920: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.storage:grp-g4d21a29422141383 TPGT: 99d TSIH: 0]####-##-##T##:##Z cpu24:2099084)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Conn [CID: 0 L: 10.68.248.232:44597 R: 10.68.248.225:3260]####-##-##T##:##Z cpu62:2176543)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:643: vmhba64:CH:1 T:0 L:0 : Task mgmt "Abort Task" with itt=0xaea (refITT=0xae8) timed out.####-##-##T##:##Z cpu70:2178881)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:643: vmhba64:CH:3 T:0 L:0 : Task mgmt "Abort Task" with itt=0x1200 (refITT=0x11fd) timed out.####-##-##T##:##Z cpu24:2099084)WARNING: iscsi_vmk: iscsivmk_StopConnection:738: vmhba64:CH:3 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)####-##-##T##:##Z cpu24:2099084)WARNING: iscsi_vmk: iscsivmk_StopConnection:739: Sess [ISID: 00023d000004 TARGET: iqn.2007-11.com.storage:grp-g4d21a29422141383 TPGT: 99d TSIH: 0]####-##-##T##:##Z cpu24:2099084)WARNING: iscsi_vmk: iscsivmk_StopConnection:740: Conn [CID: 0 L: 10.68.248.232:41050 R: 10.68.248.228:3260]####-##-##T##:##Z cpu70:2178881)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:643: vmhba64:CH:3 T:0 L:0 : Task mgmt "Abort Task" with itt=0x1201 (refITT=0x11fd) timed out.####-##-##T##:##Z cpu70:2178881)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...####-##-##T##:##Z cpu45:2098275)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...####-##-##T##:##Z cpu45:2098275)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...####-##-##T##:##Z cpu45:2098275)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...
Resolution
To resolve the issue, we have the following choices:
- Change the MTU size on the VMkernel NIC(s) and Virtual Switch configured for iSCSI traffic to 1500.
- Ensure MTU size of 9000 is at the following (if not, update it)
a. ESXi side: VMkernel NIC(s) and Virtual Switch configured for iSCSI traffic
b. Switch side: Relevant ports on the switch (Fabric Interconnects usually require MTU of 9216 as opposed to traditional Switches that will need MTU of 9000)
c. Storage Side: Relevant storage array data ports
More information regarding iSCSI connectivity troubleshooting is available at Troubleshooting ESXi connectivity to iSCSI arrays using software initiators
Feedback
