Cloud SWG logs appear out of order when downloaded from SyncAPI using curl

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Using SyncAPI to download Cloud SWG access logs from our production tenant to Splunk and everything works fine.

Created a new TEST Cloud SWG tenant and validating the SyncAPI downloads from this tenant using curl, but seeing timing issues.

When downloading the logs, I can see log entries for 10:41am UTC appearing before log entry for 10:32am UTC as shown below:

Environment

Cloud SWG.

SyncAPI.

Cause

Working as designed.

Resolution

This is working as designed. When a SIEM digests the log entries and builds the events, they are automatically sorted in chronological order.

Without a SIEM ingesting and sorting out the data, the log entries returned may be out of order for performance reasons. Curl has no ability to re-order (can be done post download with sorting tools), and adding sorting to the back end would slow down the rendering of the data.

One needs to consider the following key points regarding reporting and the Cloud SWG envronment:

• Cloud SWG service includes hundreds of Proxy SGs worldwide that are all proxying traffic, and any single customer can have traffic flowing through many of those proxies.  When the log entries from these different Proxy SGs are merged into a single stream, they aren't sorted by date. 
• Each individual Proxy SG is breaking up the log data and writing it to different reporting infrastructure components. When SyncAPI eventually gets this data, it avoids re-ordering it for performance reasons.
• When the Sync API requests are handled at the back end, it does so in a multi-threaded fashion and searches many files in parallel.  As each thread finds matching log lines, they are immediately sent to the client.