FC HBA Fabric Logouts and Path Disconnects during ToR Switch Maintenance

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

• ESXi hosts experience intermittent storage connectivity issues or session teardowns on HBAs during Top of Rack Switch Maintenance.

• Devices may enter an APD (All Paths Down) state on ESXi hosts.

Environment

 VMware vSphere ESXi 8.x

Cause

• This can occur if the storage fabric triggers Registered State Change Notifications (RSCN) during Top of Rack switch maintenance.
  
  
• An RSCN is a standard Fibre Channel mechanism used by switches to notify end devices (ESXi hosts) of a topology change within the fabric.
  
  
• When the ESXi HBA driver receives this notification, it is designed to initiate a session teardown (port logout) and re-discovery to ensure it has an accurate map of the fabric.
  
  
• This behavior is an expected reaction to external fabric events rather than an internal ESXi or driver defect.

ESXi: /var/run/log/vmkernel.log 
####-##-##T##:##:##.###Z In(182) vmkernel: cpu64:2098295)ql_fcoe:vmhba#:RecvRSCN:2365:Info: received RSCN for Fabric(0x4324c196e2a0): 010608
####-##-##T##:##:##.###Z In(182) vmkernel: cpu64:2098295)ql_fcoe:vmhba#:StartGPN_FF_Targets:3407:Info: Called.

####-##-##T##:##:##.###Z In(182) vmkernel: cpu64:2098295)ql_fcoe:vmhba#:StartPortLogout:1236:Info: Sess 0x4324c1975fb0 port_id <port_id>
####-##-##T##:##:##.###Z In(182) vmkernel: cpu64:2098295)ql_fcoe:vmhba#:CancelExchangeHandling:196:Info: Enter for Sess = 0x4324c1975fb0
####-##-##T##:##:##.###Z In(182) vmkernel: cpu0:2098509)qedf:vmhba#:qedfc_cleanup_rport:1110:Info: ST(RPORT): DISABLED C_ID[0x6]:P_ID[0x10000]:T_ID[6]

Resolution

Since the adapter is responding to unexpected external fabric triggers at the time of the switch maintenance, the resolution requires investigating the physical infrastructure.

1. Identify Timestamps: Note the exact UTC timestamps of the RecvRSCN events in the vmkernel.log.
2. Vendor Engagement: Engage the fabric vendor to perform a deep-dive analysis of the switch and storage array logs to identify why RSCN packets are being broadcast to the hosts.