Removing uplinks from an Uplink Profile causes incorrect TEP order and some VMs lose connectivity
Article ID: 345785
Updated On:
Products
VMware NSX Networking
Issue/Introduction
This is meant to provide guidance on the workflow required to reduce the number of uplinks in an Uplink Profile for NSX versions prior to 3.2.2.
Symptoms:
- After reducing the number of uplinks in an Uplink Profile, some VMs lose network connectivity.
- Adding the uplinks back to the Uplink Profile restores network connectivity.
- Logging in /var/run/log/nsx-syslog.log shows an update with incorrect TEP order.
Example logging with correct order for the 4 TEPs, vmk10, vmk11, vmk12, vmk13:
2022-11-03T23:19:51Z nsx-opsagent[6348138]: NSX 6348138 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="6348171" level="INFO"] Upgrading - restoring backup HostConfigMsg [host_switches { switch_id: "<UUID>" host_switch_profiles { uplink_profile { teaming_policy: LOADBALANCE_PORT_ID active { type: PHYSICAL_NIC name: "uplink-1" } active { type: PHYSICAL_NIC name: "uplink-2" } active { type: PHYSICAL_NIC name: "uplink-3" } active { type: PHYSICAL_NIC name: "uplink-4" } } } pnics { device_name: "uplink1" uplink_name: "uplink-1" } pnics { device_name: "uplink2" uplink_name: "uplink-2" } pnics { device_name: "uplink3" uplink_name: "uplink-3" } pnics { device_name: "uplink4" uplink_name: "uplink-4" } vteps { ip: "<vmk10 IP>" device: "" operation: ADD } vteps {
2022-11-03T23:19:51Z nsx-opsagent[6348138]: ip: "<vmk11 IP>" device: "" operation: ADD } vteps { ip: "<vmk12 IP>" device: "" operation: ADD } vteps { ip: "<vmk13 IP>" device: "" operation: ADD }
Example update where order became vmk10, vmk12, vmk11, vmk13:
2022-11-03T23:23:42Z nsx-opsagent[6348138]: NSX 6348138 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="6348167" level="INFO"] [ProcessReceiveMessages] HostConfigMsg [host_switches { switch_id: "<UUID>" host_switch_profiles { uplink_profile { teaming_policy: LOADBALANCE_PORT_ID active { type: PHYSICAL_NIC name: "uplink-1" } active { type: PHYSICAL_NIC name: "uplink-2" } active { type: PHYSICAL_NIC name: "uplink-3" } active { type: PHYSICAL_NIC name: "uplink-4" } } } pnics { device_name: "uplink1" uplink_name: "uplink-1" } pnics { device_name: "uplink2" uplink_name: "uplink-2" } pnics { device_name: "uplink3" uplink_name: "uplink-3" } pnics { device_name: "uplink4" uplink_name: "uplink-4" } vteps { ip: "<vmk10 IP>" device: "vmk10" operation: UPDATE } vteps {
2022-11-03T23:23:42Z nsx-opsagent[6348138]: ip: "<vmk12 IP>" device: "vmk12" operation: UPDATE } vteps { ip: "<vmk11 IP>" device: "vmk11" operation: UPDATE } vteps { ip: "<vmk13 IP>" device: "vmk13" operation: UPDATE }
Symptoms:
- After reducing the number of uplinks in an Uplink Profile, some VMs lose network connectivity.
- Adding the uplinks back to the Uplink Profile restores network connectivity.
- Logging in /var/run/log/nsx-syslog.log shows an update with incorrect TEP order.
Example logging with correct order for the 4 TEPs, vmk10, vmk11, vmk12, vmk13:
2022-11-03T23:19:51Z nsx-opsagent[6348138]: NSX 6348138 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="6348171" level="INFO"] Upgrading - restoring backup HostConfigMsg [host_switches { switch_id: "<UUID>" host_switch_profiles { uplink_profile { teaming_policy: LOADBALANCE_PORT_ID active { type: PHYSICAL_NIC name: "uplink-1" } active { type: PHYSICAL_NIC name: "uplink-2" } active { type: PHYSICAL_NIC name: "uplink-3" } active { type: PHYSICAL_NIC name: "uplink-4" } } } pnics { device_name: "uplink1" uplink_name: "uplink-1" } pnics { device_name: "uplink2" uplink_name: "uplink-2" } pnics { device_name: "uplink3" uplink_name: "uplink-3" } pnics { device_name: "uplink4" uplink_name: "uplink-4" } vteps { ip: "<vmk10 IP>" device: "" operation: ADD } vteps {
2022-11-03T23:19:51Z nsx-opsagent[6348138]: ip: "<vmk11 IP>" device: "" operation: ADD } vteps { ip: "<vmk12 IP>" device: "" operation: ADD } vteps { ip: "<vmk13 IP>" device: "" operation: ADD }
Example update where order became vmk10, vmk12, vmk11, vmk13:
2022-11-03T23:23:42Z nsx-opsagent[6348138]: NSX 6348138 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="6348167" level="INFO"] [ProcessReceiveMessages] HostConfigMsg [host_switches { switch_id: "<UUID>" host_switch_profiles { uplink_profile { teaming_policy: LOADBALANCE_PORT_ID active { type: PHYSICAL_NIC name: "uplink-1" } active { type: PHYSICAL_NIC name: "uplink-2" } active { type: PHYSICAL_NIC name: "uplink-3" } active { type: PHYSICAL_NIC name: "uplink-4" } } } pnics { device_name: "uplink1" uplink_name: "uplink-1" } pnics { device_name: "uplink2" uplink_name: "uplink-2" } pnics { device_name: "uplink3" uplink_name: "uplink-3" } pnics { device_name: "uplink4" uplink_name: "uplink-4" } vteps { ip: "<vmk10 IP>" device: "vmk10" operation: UPDATE } vteps {
2022-11-03T23:23:42Z nsx-opsagent[6348138]: ip: "<vmk12 IP>" device: "vmk12" operation: UPDATE } vteps { ip: "<vmk11 IP>" device: "vmk11" operation: UPDATE } vteps { ip: "<vmk13 IP>" device: "vmk13" operation: UPDATE }
Cause
The Management plane fetches existing TEPs from a hash map. Previously, the TEP order was not properly maintained in that hash map when making changes to Uplink Profile.
Resolution
This issue is resolved in NSX version 3.2.2.
Workaround:
To reduce uplink count in Uplink Profile on versions before 3.2.2:
1. Place host into Maintenance Mode
2. Configure a new Uplink Profile with the reduced uplink count
3. Uninstall NSX from the host
4. Reconfigure NSX on the host with the updated Uplink Profile
Workaround:
To reduce uplink count in Uplink Profile on versions before 3.2.2:
1. Place host into Maintenance Mode
2. Configure a new Uplink Profile with the reduced uplink count
3. Uninstall NSX from the host
4. Reconfigure NSX on the host with the updated Uplink Profile
Additional Information
Impact/Risks:
If TEP order is atypical (ex: vmk10, 12, 11, 13) when reducing the uplink count in an Uplink Profile, VMs now using TEPs that were expected to have been deleted (vmk12 in above example when going from 4 uplinks to 2 uplinks) lose network connectivity.
If TEP order is atypical (ex: vmk10, 12, 11, 13) when reducing the uplink count in an Uplink Profile, VMs now using TEPs that were expected to have been deleted (vmk12 in above example when going from 4 uplinks to 2 uplinks) lose network connectivity.
Feedback
Yes
No
Powered by
