Excessive Registered State Change Notifications (RSCNs) being reported in vmkernel.log points to physical fabric issue

Products

• VMware vSphere ESXi

Issue/Introduction

Registered State Change Notifications (RSCNs) are not an issue by themselves as they are designed to let other members of a fabric zone know where there has been a change but if excessive amount of RSCN messages are seen in /var/log/vmkernel.log, this can be in indictor that the fabric is not healthy and needs to be investigated.

 

Example of RSCN  messaging in /var/log/vmkernel.log

 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3886: RSCN payload_len: 0x800 page_len: 0x4
 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xbd0000
 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xbd0020
 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xbd0040
 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xbd0060
 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xbd0120
 YYYY-MM-DDTHH:MM:SS.432Z In(182) vmkernel: cpu73:2098051)nfnic: <2>: INFO: fdls_process_rscn: 3905: RSCN received for num_ports: 0 payload_len: 2048 page_len: 4
 YYYY-MM-DDTHH:MM:SS.466Z In(182) vmkernel: cpu80:2098044)nfnic: <1>: INFO: fdls_process_rscn: 3886: RSCN payload_len: 0x800 page_len: 0x4
 YYYY-MM-DDTHH:MM:SS.466Z In(182) vmkernel: cpu80:2098044)nfnic: <1>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xe30000
 YYYY-MM-DDTHH:MM:SS.466Z In(182) vmkernel: cpu80:2098044)nfnic: <1>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xe30020
 YYYY-MM-DDTHH:MM:SS.466Z In(182) vmkernel: cpu80:2098044)nfnic: <1>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xe30040
 YYYY-MM-DDTHH:MM:SS.466Z In(182) vmkernel: cpu80:2098044)nfnic: <1>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xe30060
 YYYY-MM-DDTHH:MM:SS.466Z In(182) vmkernel: cpu80:2098044)nfnic: <1>: INFO: fdls_process_rscn: 3896: RSCN for port id: 0xe30120

 

 

 

Environment

VMware ESXi (All Versions)

Cause

These messages occurring on the same port IDs across multiple hosts means that there is a member of the fabric zone that is performing a lot of fabric logout/logins. As a result, the array target(s) have been disconnecting from the initiators in the ESXi host and reconnecting over and over again.  This points to a SAN fabric health issue.

Resolution

We see this issue most often due to a flapping port, bad SFP or low light condition on fabric switches. As these issues originate outside of the ESXi host, we would recommend to have you reach out to your storage team/fabric switch team and/or vendors to investigate.