Sample script to download files via the Cloud SWG Sync API

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

The sync-API documentation provides some sample commands that we can extend here, however, we have customers or partners who benefit from a more detailed download tool, to get started with Sync-API or to validate that Sync-API access from a Splunk server (or other SIEM system) is fully operational.

Resolution

Using the Syn-API documentation (see Additional Information section below for references) you can quickly get started downloading WSS Access logs using the sync-api.

Here is a command set that can be used to download files in the Windows command line (with random GUIDs in place of valid API keys):

@echo off
set api_usr=9dadd73c-b434-459a-82e4-d85154460046
set api_pwd=6f510f03-db1f-4dd5-97ed-9c54ddb0df5d

set outfile=20220914T1200.zip

set starttime=1663113600000
set endtime=1663160400000

set uri="https://portal.threatpulse.com/reportpod/logs/sync?startDate=%starttime%&endDate=%endtime%&token=none"

echo %uri%

curl -H "X-APIUsername: %api_usr%" -H "X-APIPassword: %api_pwd%" -o %outfile% %uri%

And here is the same command set tailored to Linux shell:

api_usr=9dadd73c-b434-459a-82e4-d85154460046
api_pwd=6f510f03-db1f-4dd5-97ed-9c54ddb0df5d

outfile=20220914T1200.zip

starttime=1663113600000
endtime=1663160400000

uri="https://portal.threatpulse.com/reportpod/logs/sync?startDate=$starttime&endDate=$endtime&token=none"

echo $uri

curl -H "X-APIUsername: $api_usr" -H "X-APIPassword: $api_pwd" -o $outfile $uri

The above code works well but there are inherent limitations to them:

• start and end times have to be provided in epoch format which is not user friendly
• any response will be written using the outfile name
• retries are not handled nor any http error or authentication failure

In order to address the above a sample python script is attached to this KB (with a .doc extension).

The sync-API-tool ensures the downloaded file is saved in chunks (so the entire file is not written into memory before being saved) and can handle multiple tenants (i.e. the X-sync-token value is stored in a text file as "tenant.token" where the tenant is the value provided with the command --tenant.

Here are some command line samples:

# Sample command line invocations (with an invalid password ;-)
#
#python sync-api-tool-3.py --user b0090c3f-7e4e-417c-8c43-e8a4392f8d3c --key 6679a636-2e57-4ee1-bf60-20ac611f0649 --tenant 21188 --startdate "2022-09-00 00:00:00" --enddate "2022-09-30 00:00:00"
#python sync-api-tool-3.py --user b0090c3f-7e4e-417c-8c43-e8a4392f8d3c --key 6679a636-2e57-4ee1-bf60-20ac611f0649 --tenant 21188 --startdate "2022-06-00 00:00:00"
#python sync-api-tool-3.py --user b0090c3f-7e4e-417c-8c43-e8a4392f8d3c --key 6679a636-2e57-4ee1-bf60-20ac611f0649 --tenant 19432 --startdate "2022-10-01 00:00:00"
#python sync-api-tool-3.py --user b0090c3f-7e4e-417c-8c43-e8a4392f8d3c --key 6679a636-2e57-4ee1-bf60-20ac611f0649 --tenant 22054 --startdate "2021-11-01 00:00:00"

Additional Information

Use the Sync API to Get Near-Real-Time Log Data