Packet Flow in Vmware AVI Load Balancer when Tunnel Mode is Enabled

Products

• VMware Avi Load Balancer

Issue/Introduction

This document outlines the procedure for analysing packet captures from a Virtual Service (VS) and Service Engines (SEs) operating in SE_TUNNEL mode. This mode involves traffic encapsulation and forwarding between SEs, requiring careful analysis to trace connection flow.

Scenario:

• Primary Service Engine (PSE) MAC: PSE_MAC 
• Secondary Service Engine (SEC) MAC: SEC_MAC
• Virtual Service IP (VS): VS_IP
• Client IP (CLIENT): CLIENT_IP
• Client MAC (CMAC): CMAC

Environment

All Cloud Environments where se_tunnel mode is enabled.

Cause

Objective: This is to depict that though the Virtual service pcap shows packets landing directly on secondary SE, but it is tunneled through Primary SE in se_tunnel mode.

 

Resolution

 

Procedure:

1. Simultaneous Packet Capture:

• Initiate simultaneous packet captures on:
  • The Virtual Service (VS) interface.
  • Both the Primary Service Engine (PSE) and Secondary Service Engine (SEC) interfaces.
• Ensure the captures cover the duration of the client connection to be analyzed.

2. Virtual Service (VS) Packet Capture Analysis:

• Connection Handling by Secondary SE:
  • If the connection is handled by the secondary SE (SEC), the VS packet capture will show:
    • Source MAC: CMAC (Client MAC)
    • Destination MAC: SEC_MAC (Secondary SE MAC)
    • This indicates the client's initial request is directed towards the secondary SE.

3. Primary Service Engine (PSE) Packet Capture Analysis:

• Initial Packet Arrival:
  • Trace the same connection on the primary SE (PSE) packet capture using the client's source port.
  • Observe the connection landing on the primary SE.
  • Sequence Number Validation:
    • Validate the captured traffic by comparing the TCP sequence numbers between the VS capture and the SE captures. This will ensure that the traffic being analyzed is part of the same TCP flow.
• Source and Destination MAC:
  • The PSE will initially receive the packet with:
    • Source MAC: CMAC (Client MAC)
    • Destination MAC: PSE_MAC (Primary SE MAC)
• MAC Address Modification:
  • The PSE then modifies the destination MAC address from PSE_MAC to SEC_MAC.
• Encapsulation and Forwarding:
  • The PSE encapsulates the packet within an Ethernet frame and forwards it to the secondary SE (SEC).

4. Secondary Service Engine (SEC) Packet Capture Analysis:

• Decapsulation and Processing:
  • The SEC receives the encapsulated packet, removes the encapsulation, and processes the traffic.
  • Destination MAC:
    • At this point, the destination MAC address seen on the secondary SE would be SEC_MAC.
• Response Traffic:
  • Upon processing the traffic, the SEC generates a response.
  • Encapsulation and Forwarding:
    • The SEC sets the destination MAC address in the encapsulated frame to PSE_MAC and sends it back to the primary SE.

5. Primary Service Engine (PSE) Packet Capture (Response):

• Decapsulation and Client Forwarding:
  • The PSE receives the encapsulated response from the SEC, removes the encapsulation.
  • Destination MAC:
    • The PSE sets the destination MAC address in the packet to CMAC (Client MAC) and forwards it to the client.

Key Observations:

• SE_TUNNEL mode involves encapsulation and MAC address modification for traffic forwarding between SEs.
• Packet captures from the VS and both SEs are essential for tracing connection flow.
• TCP sequence number analysis is crucial for validating packet correlation.
• Understanding the MAC address changes performed by the SEs is critical to understanding the traffic flow.