LACP Port Channel flaps and goes down, between ProxySG and Cisco Nexus 9000

Products

ProxySG Software - SGOS ASG-S200 ASG-S400 ASG-S500

Issue/Introduction

The link aggregation between the proxySG and the cisco nexus 9K switches no longer works. Several error messages comE up:

2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-SPEED: Interface Ethernet1/15, operational speed changed to 1 Gbps
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/15, operational duplex mode changed to Full
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/15, operational Receive Flow Control state chang
ed to off
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/15, operational Transmit Flow Control state chan
ged to off
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-SPEED: Interface port-channel115, operational speed changed to 1 Gbps
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-IF_DUPLEX: Interface port-channel115, operational duplex mode changed to Full
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel115, operational Receive Flow Control state ch
anged to off
2022 Apr 4 11:59:04 xxxxxxxxxxxx %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel115, operational Transmit Flow Control state c
hanged to off
2022 Apr 4 11:59:08 xxxxxxxxxxxx %ETH_PORT_CHANNEL-5-PORT_UP: port-channel115: Ethernet1/15 is up
2022 Apr 4 11:59:08 xxxxxxxxxxxx %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel115: first operational port changed from none to Ether
net1/15
2022 Apr 4 11:59:08 xxxxxxxxxxxx %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel115,bandwidth changed to 1000000 Kbit
2022 Apr 4 11:59:08 xxxxxxxxxxxx %ETHPORT-5-IF_UP: Interface Ethernet1/15 is up in mode access
2022 Apr 4 11:59:08 xxxxxxxxxxxx %ETHPORT-5-IF_UP: Interface port-channel115 is up in mode access
2022 Apr 4 11:59:08 xxxxxxxxxxxx %STP-6-PORT_RANGE_ADDED: Interface port-channel115 added to vlan=1188 with cost 20000, priority 128
, link-type P2p
2022 Apr 4 11:59:08 xxxxxxxxxxxx %STP-6-PORT_RANGE_ROLE: new_role=designated interface=port-channel115 vlan=1188
2022 Apr 4 11:59:08 xxxxxxxxxxxx %STP-6-PORT_RANGE_STATE: new_state=blocking interface=port-channel115 vlan=1188
2022 Apr 4 11:59:08 xxxxxxxxxxxx %STP-6-PORT_RANGE_STATE: new_state=forwarding interface=port-channel115 vlan=1188
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel115 is down (No operational mem
bers)
2022 Apr 4 11:59:09 xxxxxxxxxxxx %STP-6-PORT_RANGE_DELETED: Interface port-channel115 removed from vlan=1188
2022 Apr 4 11:59:09 xxxxxxxxxxxx %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel115 vlan=1188
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel115: first operational port changed from Ethernet1/15
to none
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel115: Ethernet1/15 is down
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel115,bandwidth changed to 100000 Kbit
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/15 is down (Initializing)
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel115 is down (No operational mem
bers)
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-SPEED: Interface port-channel115, operational speed changed to 1 Gbps
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-IF_DUPLEX: Interface port-channel115, operational duplex mode changed to Full
2022 Apr 4 11:59:09 xxxxxxxxxxxx %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel115, operational Receive

Resolution

For the messages reported in the case description, please refer to the below.

"operational Receive Flow Control state changed to off"

From checks done, we see that the above message is linked with the Nexus 9000. This would be because of either a physical cabling issue or because the device connecting to the nexus gear was not negotiating properly and had to be hardcoded to 1000/Full. From the Cisco community source, it's recommended that the cabling be checked, first, and then hardcode the port, for sure, to ensure a match with the port speed. With LACP, everything has got to match, on both ends of the aggregation. This isn't a ProxySG message/issue and is not linked with the SGOS upgrade reported.

Ref. doc.: https://community.cisco.com/t5/server-networking/nexus-port-issues/td-p/2490586

"operational duplex mode changed to Full"

We see that the above message is linked with a Cisco Bug CSCvo20434, with Nexus 9000, and you may want to contact Cisco to report this, for their investigation. This isn't a ProxySG message/issue and is not linked with the SGOS upgrade reported.

Ref. doc.: https://quickview.cloudapps.cisco.com/quickview/bug/CSCvo20434

"operational Transmit Flow Control state changed to off"

For the above message, we have seen it reported for the Cisco Nexus 3172T, and you may want to check this up with Cisco, for Nexux 9000.

Ref. doc.: https://community.cisco.com/t5/switching/nexus-interface-traffic-keep-droping/td-p/3089830

"first operational port changed from none to Ethernet1/15"

The above message is linked with a Cisco Bug CSCvc51557, where the First Operational port changes of Nexus port-channel changes without FOP going down. Again, this isn't a ProxySG issue/message and you may have to check this up with Cisco, for the necessary intervention.

Ref. doc.: https://quickview.cloudapps.cisco.com/quickview/bug/CSCvc51557

"IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN"

The above message is, clearly, one coming from the Cisco Nexus 9000 Series NX-OS System. See the explanation/recommendation below, from Cisco.

Explanation: This is a PortChannel interface and all its members are operationally down. [optionally if the port belongs to port-channel, the port-channel number to which the port belongs will be displayed and if the interface description is configured, the interface description is displayed]

Recommended Action: Enable at least one of the PortChannel's members

https://knowledge.broadcom.com/external/article/169269/lacp-aggregation-on-a-proxysg-or-advance.html

For guidance on the expected LACP implementation on the side of the ProxySG, please refer to the guidance detailed in the Tech. Article with the URL below. Once this is correctly, and fully, implemented on the ProxySG, the ProxySG side of the LACP implementation is done. Please utilize this guide to check the related configurations on the ProxySG. Where there are inconsistencies, utilize the same guidance, to correct the config.

The LACP-related errors/messages reported are all linked with the Cisco Nexus 9000 device, with some being known bugs, and not related to the ProxySG.