search cancel

Notes on some WSS syncapi best practices

book

Article ID: 174730

calendar_today

Updated On:

Products

Web Security Service - WSS

Issue/Introduction

What to expect when deploying the SyncAPI on multiple clients for a single Threatpulse tenant/account.

Client receives a trailer status of "more"

X-sync-token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
X-sync-status: more
 

On next download, the client receives an empty file with status of "done"

X-sync-token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
X-sync-status: done

Cause

This issue is likely to arise when multiple clients are competing against each other, which can induce a couple of problems:

  1.)  At the least, the clients would consume double the bandwidth to download duplicate copies of logs.

  2.)  If one of those clients falls behind for a period of time and then attempts to catch up, the clients can end up competing.

          2a.)  If client-1 falls behind, and client-2 maintains current logs, when client-1
                   tries to catch up, it can receive a 'done' response from the server, when more
                   data is expected.  For example:
 
                  Client-1 while catching up, receives a log download with a response of 200 and trailer status of "more".
 
                  Client-2 while downloading normally, receives it's last hours worth of logs, with a trailer status of "done".
 
                  Client-1 makes a request for the next hours worth of logs, based on the "more" status it just received.
   
                  Client-1 is now served an empty file reflecting a "done" trailer, even though it was just told "more" in the response to its last request.
 
This is normal behavior and would not occur if using a single client to download logs via SyncAPI.
 
The issue above is not related to throttling an over aggressive client; If the server was throttling the client, it would return an HTTP response of 429 with a trailer consisting of a "retry-after" field that indicates when the client is okay to reattempt the request. 
 
When the issue described above occurs, both clients would receive a 200 response indicating the server is not throttling their requests.  Examination of the response trailer in the last received file on the client would reveal a more granular description of the response (i.e. done, or more - where "more" indicates additional data is available). 
 
Note:  When the server returns a "done" status to the client, it expects no other requests to be issued within 60s of sending the last successful file.  Any premature requests by the client would result in throttling until the 60s period has lapsed. 
 
Furthermore, when the server returns a "done" status to any client relative to the portal tenant from which logs are being downloaded, it would apply to all clients.  This is what might lead to a client receiving a 200 successful response, with a status of "done" in the trailer, even though the logs are not current on that specific client.

Environment

Web Security Service - threatpulse portal account

SyncAPI - Deployed on multiple clients for downloading from the same threatpulse portal account.

Resolution

The solution is to only use a single client to download logs from the WSS via the SyncAPI.  Logs should be multiplexed internally for any further redundancy/analysis, after being downloaded via the SyncAPI.