Service Engine does not connect to Controller if the port-channel MTU is configured with any value other than 1500

Products

• VMware Avi Load Balancer

Issue/Introduction

SE does not get connected to Controller if the port-channel MTU is configured with any value other than 1500.

se_dp crash will be noticed as below with a segmentation fault

[New LWP 1963]
[New LWP 2006]
[New LWP 2007]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `se_dp: worker process 0:                          '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000008946 in ?? ()
[Current thread is 1 (Thread 0x7f7c88e66f40 (LWP 1963))]
#0  0x0000000000008946 in ?? ()
#1  0x0000560eb93d9537 in gro_tcp4_reassemble

Environment

Avi version below 30.2.3-2p2

Cause

This issue is caused due to a regression in 30.2.3-2p2 and below (on 30.2.x branch) that prevented SE's from connecting to Controller when bond interfaces had mismatched MTU's

When there's a mismatch of MTU on the SE interfaces, for an ex. bond0 (mgmt) has 1500 MTU and bond1(DataPlane) has 9000 for bare metal node (LSC) - SE does not connect to Controller due to a regression caused by a change made in 30.2.x.

To Verify bond interfaces and their MTU with ip:

ip link show bond0

ip link show bond1

Resolution

Workaround

• Modify all bond (bond0 and bond1) interfaces to MTU 1500 on bare metal where SE's are running, Set MTU (if needed): ip link set dev bond0 mtu 1500 and ip link set dev bond1 mtu 1500
• Persist change (example for RHEL/CentOS): add MTU=1500 to /etc/sysconfig/network-scripts/ifcfg-bond0 and /etc/sysconfig/network-scripts/ifcfg-bond01
• Reboot the bare metal
• After reboot: ip -brief link - verify both bond0 and bond1 report MTU 1500

Fix

This issue is fixed in versions 30.2.3-2p3, 30.2.4, 31.1.2 and 31.2.1. Refer to release notes https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-2/vmware-avi-load-balancer-release-notes/release-notes-for-vmware-avi-load-balancer-version-30-2-4.html 

On fixed releases listed above, you may use MTU > 1500 (for example, 9000) for jumbo frames provided:

• All SE interfaces (management bond and dataplane bond) are set to the same MTU.
• The hypervisor/host and upstream network switches support the chosen MTU end-to-end.
• Any intermediate devices (vSwitch, VXLAN/Geneve overlays, routers) are configured to pass the larger frames or are set to appropriate MTU/MTU-aware tunneling.

Upgrade to the recommended Avi version as per KB Software Recommendation for Avi Load Balancer