MAC learning not working on NSX segment even though it appears as enabled in the NSX Manager UI. Expected IP and MAC addresses not seen in discovered bindings list.

Products

VMware NSX

Issue/Introduction

MAC learning is enabled on a newly deployed segment using a custom MAC discovery profile. It has been confirmed in the Manager UI that MAC learning is enabled on the custom profile:

It has been confirmed that the custom MAC profile is applied to the segment:

But MAC addresses configured behind the VM vNIC are not being discovered. Due to this it is not possible to connect to the IP addresses associated with those MAC addresses. Discovered bindings can be confirmed in the NSX Manager UI or using the following commands on the host running the VM:

- Run the following command to get the 'DVPort ID' of the VM:

- nsxdp-cli vswitch instance list | grep <VM Name>

- From the nsxcli on the host use the following command to retrieve the 'DVS Name':

- get logical-switch <VNI ID or Switch UUID>

- Run the following command from the nsxcli to get the discovered IP's for the VM port:

- get ip-discovery bindings <DVS Name> <DVPort ID>

The MAC table on ESXi hosts within the cluster indicate that the inner MAC is not being learnt, which is indicated by an 'Outer MAC' of 'ff:ff:ff:ff:ff:ff', which can be verified with the following commands:

- net-vdl2 -M mac -s <DVS Name> -n <VNI ID the VM is connected to>

Using the following command it can be seen that 'MAC Learning' is set to false on the VM port:

- nsxdp-cli vswitch mac-learning port get --dvport <DVPort ID> --dvs-alias <DVS Name>

MAC Learning:                   False
Unknown Unicast Flooding:       False
MAC Limit:                      4096
MAC Limit Policy:               ALLOW

Environment

VMware NSX-T Data Center 3.2.X

Cause

There is a discrepancy between Policy and Manager API, which is leading to the 'nsx-default-mac-profile' to be realized on the logical switch that the VM is connected to.
To be more specific from the policy intent API\view the segment has the custom MAC profile configured, but the corresponding Logical Switch in the Manager view\API uses the default mac discovery profile.
MAC learning needs to be realized as enabled on the Logical Switch to allow more than one MAC address to be learnt on a port. That would explain why MAC addresses behind the VM's vNIC are not being learnt, and the reason no packets are being forwarded to it.

Resolution

This is a known issue impacting VMware NSX.

The workaround is to use the API to update the binding maps of the segment, by first removing the custom MAC profile, and then updating the binding map again with the required custom MAC profile. The detailed steps are the following:

Note: Please make sure you have a recent NSX Manager backup before proceeding.

Use the following API's to check and change the segment binding map path:

GET: https://{{nsx-mgr}}/policy/api/v1/infra/segments/<Segment-Name>/segment-discovery-profile-binding-maps/

Sample response:

{
    "mac_discovery_profile_path": "/infra/mac-discovery-profiles/<custom_mac_profile_name>",
    "resource_type": "SegmentDiscoveryProfileBindingMap",
    "id": "<UUID>",
    "display_name": "<Segment-binding-UUID>",
    "path": "/infra/segments/test_segment1/segment-discovery-profile-binding-maps/<Segment-binding-UUID>",
    "relative_path": "<Segment-binding-UUID>",
    "parent_path": "/infra/segments/<Segment-Name>",
    "unique_id": "<UUID>",
    "realization_id": "<UUID>",
    "marked_for_delete": false,
    "overridden": false,
    "_create_time": 0,
    "_create_user": "unknown",
    "_last_modified_time": 0,
    "_last_modified_user": "unknown",
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 1
}

2. Copy the response from the above API to use as the payload in the below API. But remove the custom MAC profile entry from the output, before pasting it in as the payload:

"mac_discovery_profile_path": "/infra/mac-discovery-profiles/<custom_mac_profile_name>", ----------> remove this

PUT: https://{{nsx-mgr}}/policy/api/v1/infra/segments/test_segment1/segment-discovery-profile-binding-maps/<Segment-binding-UUID> 

Sample payload:

{
   "ip_discovery_profile_path": "/infra/ip-discovery-profiles/<ip_discovery_profile_name>",
    "resource_type": "SegmentDiscoveryProfileBindingMap",
    "id": "<UUID>",
    "display_name": "<Segment-binding-UUID>",
    "path": "/infra/segments/test_segment1/segment-discovery-profile-binding-maps/<Segment-binding-UUID>",
    "relative_path": "<Segment-binding-UUID>",
    "parent_path": "/infra/segments/<Segment-Name>",
    "unique_id": "<UUID>",
    "realization_id": "<UUID>",
    "marked_for_delete": false,
    "overridden": false,
    "_create_time": 0,
    "_create_user": "unknown",
    "_last_modified_time": 0,
    "_last_modified_user": "unknown",
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 1
}

3. This will set the mac profile to the default MAC profile. Again update the segment binding map by setting it back to the correct custom profile, sample profile as below:

PUT: https://{{nsx-mgr}}/policy/api/v1/infra/segments/test_segment1/segment-discovery-profile-binding-maps/<Segment-binding-UUID>
Sample Payload:

{
    "mac_discovery_profile_path": "/infra/mac-discovery-profiles/<custom_mac_profile_name>",
    "ip_discovery_profile_path": "/infra/ip-discovery-profiles/<ip_discovery_profile_name>",
    "resource_type": "SegmentDiscoveryProfileBindingMap",
    "id": "<UUID>",
    "display_name": "<Segment-binding-UUID>",
    "path": "/infra/segments/test_segment1/segment-discovery-profile-binding-maps/<Segment-binding-UUID>",
    "relative_path": "<Segment-binding-UUID>",
    "parent_path": "/infra/segments/<Segment-Name>",
    "unique_id": "<UUID>",
    "realization_id": "<UUID>",
    "marked_for_delete": false,
    "overridden": false,
    "_create_time": 0,
    "_create_user": "unknown",
    "_last_modified_time": 0,
    "_last_modified_user": "unknown",
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 1
}

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

NSX Manager support bundles
ESXi host support bundles for hosts that are running the VM's where MAC address learning is failing
Screenshots of the MAC and IP Discovery profiles assigned to the segment, both in Policy and Manager view
Screenshot of the configuration of the segment

Handling Log Bundles for offline review with Broadcom support