BGP with MD5 encryption doesn’t work when shaping is turned on

Products

• PacketShaper

Issue/Introduction

While using BGP, many implementations of BGP include MD5 signature option for security. BGP uses TCP as a transport protocol. Every segment sent on a TCP connection for BGP uses the 16-byte MD5 digest produced by applying the MD5 algorithm to the TCP header.

PacketShaper with shaping enabled, changes the initial TCP window size in the three-way handshake,  which makes MD5 checksum to fail while the packet is received on the other side. This could also become a problem if you use a rate control policy on BGP.

Resolution

Workaround:

Use the ip passthrough command to ignore the host(s) that prevent the changing of the window size and rate control from occurring if applied to the class or inherited.

If the host is on the inside, for example 10.0.0.1, then type:

ip passthrough inside src 10.0.0.1
ip passthrough outside dst 10.0.0.1

If the host is on the outside, for example 10.0.0.2, type:

ip pass outside src 10.0.0.2
ip pass inside dst 10.0.0.2

Use the ip show command to verify you are getting hits to the filter applied.

Next, create a file called startup.cmd and copy the same commands in this file. Make sure this file is located in the root directory 9.256/.

Note: BGP peers could be using a high MTU size; PS sitting in between may not be able to handle large frames. Please make sure the MTU on the BGP peers are set to 1500.