Cloud SWG SynAPI Best Practices
search cancel

Cloud SWG SynAPI Best Practices

book

Article ID: 174730

calendar_today

Updated On: 01-22-2025

Products

Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

What to expect when deploying the SyncAPI on multiple clients for a single Cloud SWG tenant/account.

The client receives a trailer status of "more"

X-sync-token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
X-sync-status: more
 

On the next download, the client receives an empty file with the status of "done"

X-sync-token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
X-sync-status: done

Environment

Cloud Secure Web Gateway

Cloud SWG SyncAPI - Deployed on multiple clients for downloading from the same or various API credentials.

Cause

This issue is likely to arise when multiple clients are competing against each other, which can induce a couple of problems:

  1.)  At the least, the clients would consume double the bandwidth to download duplicate copies of logs.

  2.)  If one of those clients falls behind for a period of time and then attempts to catch up, the clients can end up competing.

          2a.)  If client-1 falls behind, and client-2 maintains current logs, when client-1
                   tries to catch up, it can receive a 'done' response from the server, when more
                   data is expected.  For example:
 
                  Client-1 while catching up, receives a log download with a response of 200 and trailer status of "more".
 
                  Client-2 while downloading normally, receives it's last hours worth of logs, with a trailer status of "done".
 
                  Client-1 makes a request for the next hours worth of logs, based on the "more" status it just received.
   
                  Client-1 is now served an empty file reflecting a "done" trailer, even though it was just told "more" in the response to its last request.
 
This is normal behavior and would not occur if using a single client to download logs via SyncAPI.
 
The issue above is not related to throttling an over-aggressive client; If the server was throttling the client, it would return an HTTP response of 429 with a trailer consisting of a "retry-after" field that indicates when the client is okay to reattempt the request. 
 
When the issue described above occurs, both clients would receive a 200 response indicating the server is not throttling their requests.  Examination of the response trailer in the last received file on the client would reveal a more granular description of the response (i.e. done, or more - where "more" indicates additional data is available). 
 
Note:  When the server returns a "done" status to the client, it expects no other requests to be issued within 60s of sending the last successful file.  Any premature requests by the client would result in throttling until the 60s period has lapsed. 
 
Furthermore, it would apply to all clients when the server returns a "done" status to any client relative to the portal tenant from which logs are being downloaded.  This might lead to a client receiving a 200 successful response, with a status of "done" in the trailer, even though the logs are not current on that specific client.

Resolution

The solution is only using a single client to download logs from the Cloud SWG via SyncAPI.  Logs should be multiplexed internally for further redundancy/analysis, after being downloaded via the SyncAPI.