Unable to bring newly created APIC controller cluster (Cisco ACI) into healthy state when deploying the APIC appliance VM's on ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• Customer is using Cisco ACI (Application Centric Infrastructure) SDN data center solution
• LLDP is enabled on the Virtual Distributed Switch
• After deploying the 3 APIC controller appliances, looking at the Controller Status section of the APIC UI reveals none of the three nodes shows an operational state of "Available"
• Running "show lldp neighbors" CLI command on physical switch reveals the MAC address's of the ESXi host vmnics are being reported as the neighbor Device ID's, instead of the expected APIC controller hostnames
  

  (none)# show lldp neighbors
Capability codes:
(R) Router,  (B) Bridge,  (T) Telephone,  (C) DOCSIS Cable Device
(W) WLAN Access Point,  (P) Repeater,  (S) Station,  (O) Other
Device ID                   Local Intf      Hold-time  Capability   Port ID
VcD_####9b69####             Eth1/1          120                      X3
##70.###f.#754               Eth1/7          120                      ##70.###f.#754
##70.###f.#755               Eth1/8          120                      ##70.###f.#755
##70.###f.#c28               Eth1/9          120                      ##70.###f.#c28
H####SP-S##                  Eth1/49         120         BR           Eth1/1
Total entries displayed: 5

   

Environment

VMware vSphere ESXi

Cause

LLDP discovery packets sent from APIC VM to upstream physical  switch are being filtered by the BPDU filter

Resolution

• Disable Net.BlockGuestBPDU as per BPDU Filter feature in vSphere:
  
  1. Using the vSphere Client, switch to the Hosts and Clusters view.
  2. Click the desired host from the inventory tree view in the left pane.
  3. Click the Configuration Tab and then Advanced Settings under Software.
  4. Click Net and then locate the Net.BlockGuestBPDU option.
  5. Change the value to 0, which disables BPDU filtering.
  6. Click OK.

Note: By disabling BPDU filtering you are allowing all the VM vNIC's on the ESXi host to send not only LLDP but also other BPDU's 

• After disabling BFDU filter, the output of the below command should display the hostname of the APIC controllers as the LLDP neighbors of the physical switch.
  (none)# show lldp neighbors
Capability codes:
(R) Router,  (B) Bridge,  (T) Telephone,  (C) DOCSIS Cable Device
(W) WLAN Access Point,  (P) Repeater,  (S) Station,  (O) Other
Device ID             Local Intf      Hold-time  Capability   Port ID
VcD_####9b69####       Eth1/1          120                     X3
H####APIC1             Eth1/7          120                     eth2-1
H####APIC3             Eth1/9          120                     eth2-1
H####SP-S##            Eth1/49         120        BR           Eth1/1
Total entries displayed: 4
• After disabling BPDU filter, the Operational state of the 3 APIC controllers as observed within the APIC UI should show as "Available".   See below screenshot:
  
  

Additional Information

Understanding the BPDU Filter feature in vSphere
Troubleshooting Denial of Service attack in virtual infrastructure cluster
Configuring advanced options for ESXi
Deploying a virtual APIC Using VMware vCenter
LLDP frames sent from APIC VM to upstream switch are being filtered by the BPDU filter (NetMisc_DropBPDUPackets).