"Snapshot Consolidation Needed" post VM backup due to FC fabric issues

Products

• VMware vSphere ESXi

Issue/Introduction

• VMs running on FC storage are backed up using snapshot-based backups
  
  
• For several VMs, after backup completes, the vSphere Client reports for the VM:
  •  "Snapshot Consolidation Needed"
    
    
• Snapshot consolidation can subsequently be initiated successfully from the vSphere Client

Environment

VMware vSphere (all versions)

Cause

This arises if there are FC fabric level issues which prevent the backup appliance or proxy from releasing VM file locks after backup, with the result that the ESXi host cannot lock files in order to complete consolidation.  

1) /var/log/vmkernel.log may report:

a) dropped frames (e.g. via HBA driver logging):

qlnativefc: vmhba1(de:0.1): qlnativefcStatusEntry:1924:(3:8) Dropped frame(s) detected (37440 of 36864 bytes)

lpfc: lpfc_handle_status:5631: vmhba6 3271: FCP cmd x12 failed <1/0> sid x1c0d00, did x1c1001, oxid x1bdf iotag x705 Dropped Frame: Act 255, Exp 138 R
eturning Host Bus Busy

b) path state in dobut warnings (following I/O failures)
vmkwarning: cpu0:2097959)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.################################" state in doubt; requested fast path state update...

2) After snapshot consolidaiton is called the VM log (/vmfs/volumes/<datastore>/<VM>/vmware.log) reports a locked flat file e.g.

vcpu-0 - [msg.disklib.lockinfo.vmfs] Locking conflict for file "/vmfs/volumes/#########-#########-###-#############/<VM>/<VMdisk>-flat.vmdk". VMFS on-disk lock owner is host with MAC address/#########-#########-###-############# and VMFS lock mode 0x2.

Resolution

Investigate and remediate the FC fabric level issues which lead to frame drops and failed I/Os/path "state in doubt".