Alert: Frequent 'PowerOn Reset Unit Attention' Alerts on hosts indicating potential SAN Connectivity or Stability Issues

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Event messages like "Frequent PowerOn Reset Unit Attentions are occurring on device naa.##########. This may indicate a storage problem. Affected datastores: <datastore>"
• fnic aborts on multiple hosts
• Entries in the vobd.log file (located at /var/log/) resembling:

[scsiCorrelator] 5062293xxxx05us: [vob.scsi.scsipath.por] Power-on Reset occurred on vmhba37:C0:T1:L200
[scsiCorrelator] 130882373xxx5us: [vob.scsi.scsipath.por] Power-on Reset occurred on naa.##############

• SAN Congestion: The SAN might experience severe congestion, leading to prolonged I/O request completion times.
• Degraded Storage Performance: The performance of virtual machines (VMs) relying on the affected datastores can be significantly impacted.
• Path failures:
  cpu7:2055)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x41244038d380, 2056) to dev "naa.###################" on path "vmhba1:C0:T1:L0" Failed: H:0xb D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:NONE
• You may see CRC errors on the connected VMhba's:
  #esxcli storage san fc stats get -A vmhbaX may show massive CRC errors as shown below:
   Adapter: vmhba#
   Tx Frames: 2656285829
   Rx Frames: 3558837439
   Lip Count: 0
   Error Frames: 763
   Dumped Frames: 0
   Link Failure Count: 0
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 0
   Invalid Tx Word Count: 0
   Invalid CRC Count: 763 ---> ESXi host shows massive CRC count
   Input Requests: 0
   Output Requests: 0
   Control Requests: 0

Environment

VMware ESXi 7.x
VMware ESXi 8.x

Cause

The messages are caused by activity outside of ESXi’s control, only the storage administrator and vendor can isolate the root cause and provide corrective actions.

Resolution

1. Investigate the SAN:
   Consult with your SAN vendor or administrator to diagnose and address the root cause of the frequent PowerOn Reset Unit Attentions.
2. Monitor Performance
   Closely monitor the performance of the affected datastores and VMs for any signs of degradation.
3. Consider Alternatives:
   If the issue persists, consider migrating VMs to alternative datastores or hosts to minimize performance impact.

Note: This issue is likely caused by a hardware or configuration problem within the SAN. Addressing the underlying issue is crucial to ensure the stability and performance of your virtualized environment.

Additional Information

The RDM LUN resets while powering on the virtual machine

How to verify the CRC errors on VMHBA / Fiber Channel adapters on ESXi hosts