search cancel

Web Security Service SyncAPI AWS to GCP migration

book

Article ID: 215075

calendar_today

Updated On:

Products

Web Security Service - WSS

Issue/Introduction

Broadcom started migrating all our Web Security Service (WSS) Cloud Reporting Service customers from AWS to GCP. After the migration, a small subset of customers using the WSS SyncAPI may find an edge scenario where some residual data will still be trickling to AWS, while the bulk of the data will flow to GCP. In this case, the WSS SyncAPI will serve the data still in AWS until it is completely drained, and not serve the data already in GCP. No data is lost in this scenario.

Environment

Web Security Service

WSS Cloud Reporting Service

Resolution

WSS SyncAPI customers encountering this problem should reset the token used in the SyncAPI to "none" ("token=none"):

Example:

curl -k -vvv "https://portal.threatpulse.com/reportpod/logs/sync?startDate=1620867600000&endDate=0&token=none" -H "X-APIUsername:<api user name>" -H "X-APIPassword:<api password>" -o WSS_log.zip

With the next SyncAPI request, a new token will be served and customers will start receiving their full data set as expected. Data from AWS will continue to migrate to GCP and will be served with normal WSS SyncAPI requests, albeit with a delay, until the WSS service completely stops sending access log data to the old system in AWS.

Note: Date parameters are GMT in milliseconds-since-1970
https://www.epochconverter.com

Timestamp in milliseconds: 1620867600000
Date and Time (GMT): Thursday, May 13, 2021 1:00:00 AM

How to access the logs missed due to this issue?

Customers can download the data that was missed by the WSS SyncAPI due to this issue by using the WSS Download API together with the sample bash script provided below for download, as described below:

Parameters:

  • APIUsername - same used for SyncAPI
  • APIPassword - same used for SyncAPI
  • Item type - day or hour
  • Inclusive start timestamp in UTC0 "yyyy-mm-dd hh:mm"
  • Inclusive end timestamp in UTC0 "yyyy-mm-dd hh:mm"

Example script execution: 

wss-download-backfill.sh <api-user-name> <api-password> hour "2021-05-13 01:00" "2021-05-13 23:00"

Output: A compressed archive containing the logs for the dates provided in the script paramters. 

Script download:

Attached to the article is a sample script. Customers are free to adapt the sample script to their needs. Once downloaded, make the script executable using the command:

chmod +x 1620938959189__wss-download-backfill.sh 

Attachments

1620938959189__wss-download-backfill.sh.zip get_app