Intermittent Host CPU spikes when using Veeam Volume Backups with Nimble Storage Array

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Intermittent CPU spikes to 100% for Host's because of leftover Nimble Snapshots. Automatic Host compliance checks, Exporting logs may also trigger the CPU spike in this scenario. We can find similiar log entries in those cases:
**vmkernel.log**
yy-mm-dd cpu18:2098344)WARNING: NMP: nmp_PathDetermineFailure:3527: Cmd (0x9e) PDL error (0x5/0x25/0x0) - path vmhba1:C0:T##16:L## device eui.######################## - triggering path failover
yy-mm-dd cpu18:2098344)WARNING: NMP: nmpCompleteRetryForPath:394: Logical device "eui.########################": awaiting fast path state update before retrying failed command again...
yy-mm-dd cpu3:9757009)WARNING: VMW_SATP_ALUA: satp_alua_issueInquiry:83: Target completed the Inquiry VPD Page 0x83 request with good status but returned junk data for path vmhba#:C0:T##:L##
yy-mm-dd cpu3:9757009)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:160: Could not get page 83 INQUIRY data for path "vmhba1:C0:T##:L##" - Failure (195887105) 
yy-mm-dd cpu3:9757009)WARNING: VMW_SATP_ALUA: satp_alua_issueInquiry:83: Target completed the Inquiry VPD Page 0x83 request with good status but returned junk data for path vmhba1:C0:T##:L##

Environment

VMware vSphere ESXi 7.0.x

Cause

This issue could be because of leftover Nimble snapshots as the process is similiar to below  :
"Veeam is asking the array to take a snapshot and online it, so it can do its backup.  During this time, if the ESXi host does a re-scan, it will see the online snapshot as a device, then when Veeam asks the array to delete the snapshot, and the array does, VMware will lose access to the device and report a PDL.  To keep this from happening, for the volumes that Veeam is backing up, you must adjust the ACL for the hosts to "Volume Only," instead of "Volume and Snapshot."  this way the host will not see the online snapshots"

The ESXi host, believing the device is still available, retries all SCSI commands indefinitely. This has an impact on the management agents such as hostd, vpxa etc, as their commands are not responded to until the device is again accessible. This causes the ESXi host to become not-responding in the vCenter Server or have performance issues due to task's getting queued. Host CPU spikes are also observed in these scenarios. Hostd in particular can be very susceptible to storage and networking related issues.

Resolution

Please do engage the Storage array vendor for resolution.
Link for reference:
https://helpcenter.veeam.com/docs/backup/vsphere/nimble_add_name.html?ver=120