Broadcom NICs installed on Cisco UCS servers disconnect randomly

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Broadcom NICs disconnect randomly.
• The host goes unresponsive and rebooting is the only option.
• In some cases, the ESXi host fails with a purple diagnostic screen.
• Updating to the latest bnx2 driver does not resolve the issue.
• In the vmkernel log file, you see entries similar to:
  
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: < --- start FTQ dump --- >
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RV2P_PFTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RV2P_TFTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RV2P_MFTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: TBDR_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: TSCH_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: TDMA_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: TXP_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: TPAT_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: TAS_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RXP_CFTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RXP_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RLUP_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: COM_COMXQ_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: COM_COMTQ_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: COM_COMQ_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: CP_CPQ_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: RDMA_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: CSCH_CH_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: MCP_MCPQ_FTQ_CTL ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: CPU states:
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: 045000 mode ffffffff state ffffffff evt_mask ffffffff pc ffffffff pc ffffffff instr ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: 085000 mode ffffffff state ffffffff evt_mask ffffffff pc ffffffff pc ffffffff instr ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: 0c5000 mode ffffffff state ffffffff evt_mask ffffffff pc ffffffff pc ffffffff instr ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: 105000 mode ffffffff state ffffffff evt_mask ffffffff pc ffffffff pc ffffffff instr ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: 145000 mode ffffffff state ffffffff evt_mask ffffffff pc ffffffff pc ffffffff instr ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: 185000 mode ffffffff state ffffffff evt_mask ffffffff pc ffffffff pc ffffffff instr ffffffff
  cpu13:8738)< 3 >bnx2 0000:07:00.1: vmnic7: < --- end FTQ dump --- >

Cause

 

Resolution

This is a known issue between the PCie Express switch on Broadcom Quad port 1GbE adapter and the Cisco UCS-C240-M3S server.

 

To work around this issue, see the Cisco Defect ID CSCug65301 Broadcom 5709 on C240M3 looses connectivity to ESXi 5.0 intermittently.

If the issue recurs on latest bnx2 drivers, contact Cisco.

 

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box