NSX BME (Bare Metal Edge) is disconnected after reboot
Article ID: 319973
Updated On:
Products
Issue/Introduction
DP does not start (e.g. get logical-routers does not show any routers), another possible symptom is that some tunnel connections are down.
Environment
Cause
When a BME boots up, LINUX kernel uses MAC address of ethernet NIC to name interfaces (e.g. eth0, eth1, eth2, etc). It saves the mapping between MAC and interface name in a file called persistent firewall rule file (/etc/udev/rules.d/70-nsx-persistent-net.rules). Because of a race condition, bond0 interface may use the MAC address of a different member interface as the last reboot. Kernel cannot assign correct MAC to interface name mapping.
Relevant log location
The best way to identify the problem is to look at the persistent firewall rule file (/etc/udev/rules.d/70-nsx-persistent-net.rules). If the problem is hit, there are multiple interface with the same MAC address. Below is an example:SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="##:##:##:##:##:b0ace, NAME="eth0" ===> same macSUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}==abe##:##:##:##:##:57", NAME="eth1"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}==abe##:##:##:##:##:5e", NAME="eth2"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="##:##:##:##:##:b0", NAME="eth3" ===> same mac
In NSX-T version 3.1 The duplicate MAC address is still shown in BME i.e. bond interfaces show duplicate mac addresses (#ip link | grep -A 1 bond), but DP starts properly because MAC addresses are not saved in persistent firewall rule file.
Resolution
Upgrade to 3.2.3.1 and 4.1.1.
Workaround:
Please be careful while editing the rule.
1. Find a BME running the same version.
2. Copy the following file from the BME to your local drive.
/opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py
3. Use an editor. Search for "bond_primary is not None" and insert 2 lines (make sure indentation is using spaces, not tabs): if self._bond_primary is not None: result += " bond-primary %s\n" % (self._bond_primary) if self._bond_primary != self._name: <---------------- result += " pre-up sleep 5\n" <---------------- if self._bond_miimon is not None: result += " bond-miimon %s\n" % (self._bond_miimon)
4. Run sha256 checksum on the file in step #1 or 4.root@edge:# sha256sum /tmp/interfaces.py 030d47455bf1f32244cb5f857e0bcbd25e3c3fa31d2ea6ebe6407746#### /tmp/interfaces.py
5. Send the modified script file in step # 1 or 4 AND the sha256 check to the customer.
6. Use root to copy interfaces.py to /tmp/interfaces.py on affected edge.
7. Verify the sha256 checksum of /tmp/interfacs.py. Make sure that the sha256 checksum is same as in step #5.root@edge:/# sha256sum /tmp/interfaces.py 030d47455bf1f32244cb5f857e0bcbd25e3c3fa31d2ea6ebe6407746#### /tmp/interfaces.py
8. Compare /tmp/interfaces.py and /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py.
The difference should be two lines added in step #4root@edge:/# diff /tmp/interfaces.py /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py287a288,289 if self._bond_primary != self._name: result += " pre-up sleep 5\n"
9. Log in to an edge node as root. Make a copy of the existing script.cp /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py.prior
10. Copy /tmp/interfaces.py to /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.pycp /tmp/interfaces.py /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py
11. Verify the sha256 checksum of /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.pyroot@edge:/opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network# sha256sum /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py030d47455bf1f32244cb5f857e0bcbd25e3c3fa31d2ea6ebe6407746#### /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py
12. Verify the owner and permission of /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.pyroot@edge:/opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network# ls -l /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py
-rw-r--r-- 1 root root 89940 Jan 1 2000 /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/network/interfaces.py
13. Use the MAC address of ethernet NICs and find the correct MAC to interface name mapping. Edit the persistent net rules file to correct the MAC-to-name mappings (i.e. /etc/udev/rules.d/70-nsx-persistent-net.rules)
14. Edit /etc/network/interfaces. Locate the bond secondary interface config.
Add the following line to the backup interface:pre-up sleep 5
Below is an example of /etc/network/interfaces:auto loiface lo inet loopbackauto bond0iface bond0 inet dhcp bond-mode active-backup bond-miimon 100 bond-updelay 1000 bond-slaves noneauto eth0iface eth0 inet manual bond-mode active-backup bond-master bond0 bond-primary eth0 bond-slaves eth0,eth3auto eth3iface eth3 inet manual bond-mode active-backup bond-master bond0 bond-primary eth0 bond-slaves eth0,eth3 pre-up sleep 5 <-------
From the above, bond-primary eth0 shows that eth0 is the primary interface (i.e. eth3 is the backup interface). We add pre-up sleep 5 under eth3.
15. Disable the persistent net rules script to prevent rewriting of the rules file during reboots:
systemctl disable nsx-edge-net-rules.service
16. Reboot the BME.
17. Enable the persistent net rules script after reboot:systemctl start nsx-edge-net-rules.service
== Note for version 3.1.x ==
The duplicate MAC address is still shown in BME i.e. bond interfaces show duplicate mac addresses (#ip link | grep -A 1 bond), but DP starts properly because MAC addresses are not saved in persistent firewall rule file.
In version 3.1.x PCIe addresses are used in peristent firewall rules file. For e.g.root@edge:~# more /etc/udev/rules.d/70-nsx-persistent-net.rulesSUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:3b:##.0", NAME="eth0"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:3b:##.1", NAME="eth1"
Additional Information
Feedback
