NSX transport node is in partial success state with error "Host configuration: DefaultGateway [#.#.#.#] Failed to be applied: Failed to update vxlan default gateway"

Products

• VMware NSX

Issue/Introduction

While applying a profile update to the transport node following error may be seen in NSX manager UI.

Host configuration: DefaultGateway [#.#.#.#] Failed to be applied: Failed to update vxlan default gateway: {'fault': 'InvalidArgument', 'msg': 'A specified parameter was not corrected:', 'faultMessage': []}; Ipv6DefaultGateway [] failed to be applied: Failed to update vxlan default gateway: {'fault': 'InvalidArgument', 'msg': 'A Specified parameter was not correct:', 'faultMessage':[]}

Environment

VMware NSX 4.x
VMware vSphere 8.x

 

Cause

Starting with vSphere 8.0, in both Integrated Dell Remote Access Controller (iDRAC) and HP Integrated Lights Out (ILO), when you have a USB interface enabled, vUSB or vNIC respectively, an additional standard switch vSwitchBMC with uplink vusb0 gets created on the ESXi host. This causes the NSX Transport node Policy to fail to apply.

NSX Manager   /var/log/proton/nsxapi.log -

ERROR L2HostConfigTaskExecutor3 TransportNodeAsyncUtils 5228 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8817" level="ERROR" subcomp="manager"] Some error occured when configuring host switch on host: DefaultGateway [##.##.##.1] failed to be applied:
 Failed to update vxlan default gateway: {'fault': 'InvalidArgument', 'msg': 'A specified parameter was not correct: ', 'faultMessage': []}; Ipv6DefaultGateway [] failed to be applied: Failed to update vxlan default gateway: {'fault': 'InvalidArgument', 'msg': 'A specified parameter was not correct: ', 'faultMessage': []};

ESXi var/run/log/hostd.log:

In(166) Hostd[2101656]: [Originator@6876 sub=Vimsvc.TaskManager opID=f4411598 sid=52627d## user=nsx-user] Task Created : haTask--vim.dvs.HostDistributedVirtualSwitchManager.fetchPortState-374851
In(166) Hostd[2101658]: [Originator@6876 sub=Vimsvc.TaskManager opID=f4411598 sid=52627d## user=nsx-user] Task Completed : haTask--vim.dvs.HostDistributedVirtualSwitchManager.fetchPortState-374851 Status success
In(166) Hostd[2101655]: [Originator@6876 sub=Vimsvc.TaskManager opID=f44115## sid=52627d## user=nsx-user] Task Created : haTask-ha-host-vim.host.NetworkSystem.updateNetworkConfig-374852
In(166) Hostd[2101650]: [Originator@6876 sub=Hostsvc opID=f44115## sid=52627d## user=nsx-user] UpdateNetwork with Config: (vim.host.NetworkConfig) {
In(166) Hostd[2101623]: -->    netStackSpec = (vim.host.NetworkConfig.NetStackSpec) [
In(166) Hostd[2101623]: -->       (vim.host.NetworkConfig.NetStackSpec) {
In(166) Hostd[2101623]: -->          netStackInstance = (vim.host.NetStackInstance) {
In(166) Hostd[2101623]: -->             key = "vxlan",
In(166) Hostd[2101623]: -->             ipRouteConfig = (vim.host.IpRouteConfig) {
In(166) Hostd[2101623]: -->                defaultGateway = "##.##.##.1",
In(166) Hostd[2101623]: -->             },
In(166) Hostd[2101623]: -->          },
In(166) Hostd[2101623]: -->          operation = "edit"
In(166) Hostd[2101623]: -->       }
In(166) Hostd[2101623]: -->    ],
In(166) Hostd[2101623]: --> }
In(166) Hostd[2101650]: [Originator@6876 sub=Hostsvc opID=f44115## sid=52627d## user=nsx-user] Invalid NIC order: one of more PNIC specified are not uplinked to vswitch.
In(166) Hostd[2101650]: [Originator@6876 sub=Hostsvc opID=f44115## sid=52627d## user=nsx-user] Invalid NIC teaming policy: invalid NIC order specified.
In(166) Hostd[2101650]: [Originator@6876 sub=Hostsvc opID=f44115## sid=52627d## user=nsx-user] Invalid network policy: invalid NIC teaming policy.
In(166) Hostd[2101650]: [Originator@6876 sub=Hostsvc.NetworkVmkSimulator opID=f44115## sid=52627d## user=nsx-user] Cannot set policy for vswitch [vSwitchBMC], network policy is invalid: (vim.host.NetworkPolicy) {
In(166) Hostd[2101623]: -->    security = (vim.host.NetworkPolicy.SecurityPolicy) {

Resolution

Workaround:

1: Use the following command to prevent the creation of a virtual switch vSwitchBMC and associated portgroups on the next reboot of host.

[root@esx:~] esxcfg-advcfg -s 0 /Net/BMCNetworkEnable

The value of BMCNetworkEnable is 0 and the service is disabled. It can be validated using following command -

[root@esx:~] esxcfg-advcfg -g /Net/BMCNetworkEnable
Output 'Value of BMCNetworkEnable is 0' means disabled.

2: Remove the vUSB uplink for the virtual switch vSwitchBMC on the ESXi host 

    a. Login to vCenter server, select the Esxi host.
    b. Select configure.
    c. Under "Networking" select "Virtual switches".
    d. Scroll down to the "vSwitchBMC",  select  "Manage Physical Adapters" and remove the  "vusb0" uplink.
    e. Reboot the ESXi host.

3:  Re-apply the Transport Node Profile update from NSX manager UI if reboot does not automatically apply it.

Additional Information

Please refer to the KB article below:

ESXi creates standard switch vSwitchBMC with uplink vusb0 automatically after upgrading to 8.0.3 version