Host Transport Node status changing to "Switch Configuration Applying" every 5 minutes

Products

• VMware NSX

Issue/Introduction

• Impacted VMs are connected to NSX Overlay segment.
• MAC address of the vdrPort instance has been customized and contains upper case letters. 
  Note the default vdrPort MAC address is 00:50:56:56:44:52.
  Default MAC address of vdr_mac_nested is 00:50:56:56:44:53.
• In 5 minute intervals, in NSX UI, status of Host Transport Nodes may change to "Switch Configuration Applying".
• In the same timeframe, there may be unexpected latency or timeouts on routed VM traffic (traffic traversing vdrPort).
• vdrPort's PortNum will change in the same interval. This change can be observed in the output of:
  net-stats -l | grep vdrPort
• NSX Manager logs may log the line similar to the log below:
  /var/log/proton/nsxapi.log
2024-12-13T13:00:00.626Z INFO providerTaskExecutor-44 GlobalConfigProviderNsxT 8772 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] GlobalConfig Id: global-config, Revision: 3. Update GlobalConfig VdrMac from 02:50:56:56:aa:bb to 02:50:56:56:AA:BB. Update GlobalConfig VdrMacNested from 02:50:56:56:44:53 to 02:50:56:56:44:53. Syncing VDR MAC on TransportNodes...
2024-12-13T13:00:00.629Z INFO providerTaskExecutor-44 VdrMacEventUfoDao 8772 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updated to VdrMacEvent [id=GlobalConfig, sourceType=SRC_GLOBAL_CFG, sourceRevision=3]
• Use an API client of your preference to confirm the MAC address of vdrPort instance contains upper case letters:
  GET https://<Manager_IP>/policy/api/v1/infra/global-config
  e.g.:
  {
  "mtu" : 8900,
  "fips" : {
    "lb_fips_enabled" : true,
    "tls_fips_enabled" : false
  },
  "l3_forwarding_mode" : "IPV4_ONLY",
  "uplink_mtu_threshold" : 9000,
  "vdr_mac" : "02:50:56:56:AA:BB",
  "vdr_mac_nested" : "02:50:56:56:44:53",
  "allow_changing_vdr_mac_in_use" : false,
  "arp_limit_per_gateway" : 50000,
  "external_gateway_bfd" : {
    "bfd_profile_path" : "/infra/bfd-profiles/default-external-gw-bfd-profile",
    "enable" : true
  },
  "lb_ecmp" : false,
  "remote_tunnel_physical_mtu" : 9000,
  "physical_uplink_mtu" : 9000,
  "global_replication_mode_enabled" : false,
  "is_inherited" : false,
  "site_infos" : [ ],
  "resource_type" : "GlobalConfig",
  "id" : "global-config",
  "display_name" : "default",
  "path" : "/infra/global-config",
  "relative_path" : "global-config",
  "parent_path" : "/infra",
  "remote_path" : "",
  "unique_id" : "########-####-####-####-########c7ff",
  "realization_id" : "########-####-####-####-########c7ff",
  "owner_id" : "########-####-####-####-########b899",
  "marked_for_delete" : false,
  "overridden" : false,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_create_time" : 1729696108706,
  "_create_user" : "system",
  "_last_modified_time" : 1730049431534,
  "_last_modified_user" : "admin",
  "_revision" : 3
}
  
  

Environment

• VMware NSX-T Data Center prior to version 3.2.3.
• VMware NSX prior to version 4.1.1.
• VMs are connected to NSX Overlay segment.

Cause

• This is a known issue, impacting VMware NSX-T Data Center and VMware NSX deployments.
• In GlobalConfig, MAC address of the vdrPort instance is configured with uppercase characters.
• Upper case in GlobalConfig will cause an inconsistency when the MAC address is compared with the MAC address stored in the database (as the actual vdrPort is stored and created with MAC address which contains lowercase characters).
• This inconsistency is detected by internal checks that are run every 5 minutes.
• While processing the GlobalConfig, NSX will persist MAC address of vdrPort instance with lowercase characters, instead of upper case defined in GlobalConfig.
• As the test is case sensitive, this will cause a mismatch in vdrPort's MAC address, and initiates update of vdrPort's MAC address.
• While the vdrPort is being re-configured (for this purpose, it needs to be re-created), the vdrPort instance is disconnected and re-connected every 5 minutes.
• On the VM side, this will manifest in increased latency, or traffic drops.
  ICMP may return type 3, code 1 message "Destination Host Unreachable".

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.3 and VMware NSX 4.1.1, available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the KB article Download Broadcom products and software.

For a workaround:

1. Review GlobalConfig on NSX Manager:
   GET https://<Manager_IP>/policy/api/v1/infra/global-config
2. If there are uppercase characters in vdrPort's MAC address (either in vdr_mac or vdr_mac_nested), make the following changes:
   1. change these to lowercase characters (use the same character in lowercase)
   2. set the below property from false to true:
      "allow_changing_vdr_mac_in_use": true
Below is a sample payload (note the MAC is not changing, we have only converted the uppercase letters to lowercase letters):
        {
   "vdr_mac" : "02:50:56:56:aa:bb",
   "allow_changing_vdr_mac_in_use" : true
  }
3. Do a PATCH with the correct MAC address (only use the line that you intend to change).
   PATCH https://<Manager_IP>/policy/api/v1/infra/global-config
4. (Optional) Use the PATCH API discussed in step 3. again to revert ""allow_changing_vdr_mac_in_use": true" back to false:
   {
  "allow_changing_vdr_mac_in_use": false
}

Additional Information

• NSX REST API Guide: Read global configuration.
• NSX REST API Guide: Update the global configuration.