Users cannot connect to internet sites using Cloud SWG using IPSec after office holiday shutdown

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Users accessing internet via Cloud SWG service using IPSEC access method.

Immediately after returning to work following a company shutdown, no users could access any internet sites.

Users reporting connectivity errors accessing all web sites and services.

IPSec tunnel to Cloud SWG appears to be up with outgoing packets, but no responses appear to be coming back.

Environment

IPSEC firewall.

Cloud SWG.

Cause

Load balancer persistence entry for ESP traffic removed after 4 days of no activity.

Resolution

Make sure that application level traffic is sent across the IPSEC tunnel all the time using healthcheck probes.

Load balancers are in the process of getting upgraded (completion May 2023), whereby ESP and IKE traffic will correlated for persistence reasons, and this issue will not be seen even if no application level ESP traffic generated for a period of time longer than the persistent timeout.

 

Additional Information

IPSEC tunnels into Cloud SWG typically trigger two sets of connections:

a) IKE traffic where probes (DPD) continuously check the state of the tunnel. This is typically over UDP 500.

b) ESP traffic which includes the encrypted Web requests from clients. This is a seperate session over UDP 4500 (Nat traversal use case) or using ESP protocol.


Ideally healthcheck probes should be visible over these two connections using DPD (IKE), or IP SLA (icmp or HTTP layer within ESP). When this happens, the load balancer fronting these two connections will maintain it's state table.

If there is no IP SLA or Application level probes, and no user traffic is generated for a long period of time (greater than 4 days in this scenario), then the load balancer will remove this entry from it's persistence table. When ESP traffic subsequently arrives after greater than 4 days of inactivity, a new connection comes in and the load balancer may send it to a different back end Cloud SWG pod than the IKE connection. When this happens, no web responses will be seen across the tunnel from the client's perspective. Long time inactivity on IPSec tunnel can result into split persistence records on load balancer.