ESXi host lose connectivity to NetApp datastores on specific workload patterns

Products

• VMware vSphere ESXi

Issue/Introduction

• Datastores from NetApp intermittently disconnect.
• VMs become unresponsive intermittently 
• ESXi host becomes unresponsive due to loss of datastore connectivity 
• Datastore are provisioned from NetApp Array running Ontap version 9.16.1P2
• vmkernel logs may show below events
  
  WARNING: ScsiDeviceIO: 13030: READ CAPACITY on device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from Plugin "NMP" failed. I/O error
  
  
• scsi commands fail with sense data H:0x7 D:0x0 P:0x0 indicating a storage initiator error
• Symptoms may also include datastore heartbeat timeouts, lost access to volumes events. 

Environment

VMware vSphere ESXi 8.x connected to NetApp storage with Ontap 9.16.1

Cause

This is caused due to a bug in NetApp Ontap version 9.16.1 and default extreme-fixed QOS policy applied to the individual volumes. 

Resolution

This is not a VMware issue. Contact NetApp support to disable extreme-fixed QOS policy for all or the affected volumes. For more information refer Netapp article QOS bug