High latency on local datastore with failed task management aborts

Products

• VMware vSphere ESXi

Issue/Introduction

• Local datastore on HPE server reports very high latency (> 1 ms)
  
  
• VMs may take a long time to boot fully and have extremely poor performance
  
  
• VMware tools service on the VMs may fail to start
  
  
• ESXi /var/log/vmkernel.log repeatedly reports failed task management aborts against the device:
  
  ####-##-##T##:##:##.###Z In(182) vmkernel: cpu16:2097847)smartpqi01: pqisrc_taskMgmt:1816: TMF virt reset Issued for CMD : 2a B1:T0:L1 with req tag : 0x20f cmd tag 0x42d
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu16:2097847)WARNING: smartpqi: pqisrc_taskMgmt:1832 :[177:0.0][1,0,1]:TMF virt reset is failed with status : -1

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

The specific task management abort failures reported in the logs indicate a firmware, controller or device level issue.  

Resolution

Engage the hardware vendor for further investigation.