Troubleshooting End-to-End Traffic Flow for a Proxy Chain

Products

• ISG Proxy

Issue/Introduction

For the recommended implementation guidance, refer to the Tech. Article with the URL below.

Procedure to perform Edge SWG (ProxySG) Chaining

This Article references the troubleshooting steps for the end-to-end traffic flow from the child proxy to the parent (upstream) proxy, and to the Web Isolation tenant, and finally, to the OCS.

Environment

ISG-Proxy/SG/ASG

Resolution

Troubleshooting Steps:

Now, to troubleshoot the traffic flow from the Child Proxy to the Parent Proxy and then to the Web Isolation Cloud tenant, specific PCAP filters and analysis points are necessary to pinpoint where delays or issues occur. Below are the filters, analysis criteria, and troubleshooting steps. Ensure to collect the policy trace debug on both the child and parent (upstream) proxies, concurrently (same time) alongside the PCAPs, from both Proxies.

1. Filters for Capturing Relevant Traffic

Use these filters to focus on specific traffic for efficient troubleshooting:

a. General Traffic Between Child and Parent Proxies

• Filter for traffic between the child and parent proxies:
  
  host <Parent_Proxy_IP>
  
  
• If the child proxy is forwarding traffic to multiple upstream proxies, refine further with destination ports:
   

  host <Parent_Proxy_IP> and port 8080


  (Replace 8080 with the port used for proxy forwarding.)

b. Traffic Between Parent Proxy and Web Isolation Cloud

• To capture traffic between the parent proxy and the Web Isolation Cloud tenant:
   

  host <WI_Tenant_IP>


• If DNS resolution is enabled and the WI tenant has a hostname:
   

  host <WI_Tenant_Hostname>

c. DNS Resolution Traffic

• To verify DNS lookups for Web Isolation or destination websites:
   

  port 53


• Combine this with the target hostname:
   

  port 53 and host <Web_Isolation_Hostname>

d. Specific Website Traffic (Destination Website)

• To isolate traffic to the ultimate destination website:
   

  host <Destination_Website_IP>

e. Filter for HTTP/HTTPS Traffic

• HTTP traffic between proxies:
   

  port 80


• HTTPS traffic:
   

  port 443

f. Filter for Latency Analysis

• If you suspect delays in TCP handshakes or retransmissions:
   

  tcp.flags.syn == 1


• To check retransmissions:
   

  tcp.analysis.retransmission

   

2. What to Look for in the PCAP

a. Traffic Between Child Proxy and Parent Proxy

• Verify Forwarding: Ensure the child proxy successfully forwards traffic to the parent proxy:
  • Look for CONNECT requests (for HTTPS) or HTTP GET/POST requests.
  • Confirm the parent proxy responds with HTTP 200 or appropriate status codes.
• Latency in Acknowledgments: Check for delays between the child's request and the parent's response.

b. Traffic Between Parent Proxy and Web Isolation

• Successful Forwarding: Verify the parent proxy forwards traffic to the Web Isolation Cloud tenant:
  • Look for the destination IP or hostname of the WI tenant in the packets.
• TCP Handshake Issues: Check for:
  • Retransmissions (indicating packet loss).
  • Delays in SYN, SYN-ACK, and ACK sequences.
• DNS Resolutions: Ensure DNS queries for the Web Isolation tenant resolve correctly.
• TLS Handshake Issues: For HTTPS traffic, inspect:
  • The ClientHello, ServerHello, and certificate exchange.
  • Ensure no handshake failures or timeouts.

c. Traffic from Web Isolation to the Destination Website

• Successful Requests: Confirm the Web Isolation tenant requests resources from the destination website.
• Response Codes: Look for HTTP response codes:
  • 200 OK (Success)
  • 403 Forbidden or 404 Not Found (Blocked or unavailable resources).

d. Delays in Packet Timing

• Use Wireshark’s Time Delta feature to measure delays:
  • Go to Statistics > Conversations, and look at the Delta Time column.
  • Check for delays in request-response pairs, particularly between:
    • Child and Parent Proxy.
    • Parent Proxy and Web Isolation Cloud.

e. MTU or Fragmentation Issues

• Look for ICMP Fragmentation Needed packets if MTU mismatches are suspected:
   

  icmp.type == 3 and icmp.code == 4

f. Bandwidth or Congestion Issues

• Check for:
  • TCP ZeroWindow packets, indicating the receiver’s buffer is full.
  • Retransmissions or duplicate ACKs, indicating potential congestion.

3. Troubleshooting the Traffic Flow

Step 1: Child Proxy to Parent Proxy

• Ensure that:
  • Requests are correctly forwarded to the parent proxy.
  • No excessive delays exist between request and response.
• Look for errors such as 504 Gateway Timeout or 403 Forbidden.

Step 2: Parent Proxy to Web Isolation

• Confirm that:
  • The parent proxy resolves the Web Isolation hostname and establishes a connection.
  • TCP handshakes and TLS exchanges succeed without retransmissions or timeouts.

Step 3: Web Isolation to Destination Website

• Verify that:
  • Web Isolation successfully connects to the destination website.
  • HTTP requests are completed, and response codes (e.g., 200 OK) are appropriate.

Step 4: Measure End-to-End Timing

• Use packet timestamps to calculate delays at each stage:
  • Child Proxy → Parent Proxy
  • Parent Proxy → Web Isolation
  • Web Isolation → Destination Website

4. Tools for Analysis

• Wireshark Filters:
  • Use filters like http, tcp.analysis.retransmission, and tcp.flags.syn.
• Delta Time Display:
  • In Wireshark: View > Time Display Format > Seconds Since Previous Displayed Packet.
• Statistics:
  • Use Statistics > Conversations to summarize traffic.

Next Steps

1. Collect PCAPs at all critical points:
   • Between the child proxy and parent proxy.
   • Between the parent proxy and Web Isolation tenant.
   • From Web Isolation to the destination website.
2. Analyze using the filters and observations provided above.
3. Share findings, such as response codes, handshake issues, and DNS resolutions, for further analysis.

Additional Information

Linked Case: 36019773